Hardware implementation of windowing operations in three or more dimensions

Abstract

The invention relates to hardware implementations of windowed operations in three or more dimensions. A data processing system and method are disclosed for implementing a windowed operation in at least three traversal dimensions. The data processing system maps the windowed operation in at least three traversal dimensions to a plurality of constituent windowed operations in two traversal dimensions. The plurality of 2-D windowed operations are implemented in at least one hardware accelerator as such. The data processing system assembles the results of the constituent 2-D windowed operations to produce the result of the windowed operation in at least three traversal dimensions.

Description

Technical Field

[0001] This invention relates to performing windowing operations in a hardware accelerator. Background Technology

[0002] Windowing operations such as 2D convolution and pooling are well-known and widely used in the context of neural networks—especially convolutional neural networks (CNNs). For example, these 2D operations enable the application of neural network-based algorithms to image processing / analysis / understanding tasks. Hardware accelerators are known in the art and are optimized for performing these 2D operations.

[0003] Tensors used in CNNs that process 2-D images typically have more than two dimensions. Additional dimensions are added by factors such as the number of input channels associated with each image and the number of filters to be applied to each channel. An example tensor has four dimensions: batch size, channels, height, and width. The corresponding 2-D convolutional kernel applied to this tensor also has four dimensions: the number of output channels, the number of input channels, kernel height, and kernel width.

[0004] It is worth noting that regardless of the number of dimensions of the tensor representing the 2D signal, the windowing operations performed (e.g., convolution or pooling) are inherently two-dimensional. That is, the window (e.g., the kernel) only shifts its stride across two dimensions (i.e., the height and width dimensions). These two dimensions are said to be "traversed" by the window. Other dimensions, such as batch and channel dimensions, may also exist. Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, a series of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] A data processing system and method are disclosed for performing windowing operations in at least three traversal dimensions. The data processing system maps windowing operations in at least three traversal dimensions to multiple constituent windowing operations in two traversal dimensions. These multiple 2D windowing operations are thus implemented in one or more hardware accelerators. The data processing system assembles the results of the constituent 2D windowing operations to produce the result of the windowing operations in at least three traversal dimensions.

[0007] According to one aspect, a method is provided for performing windowing operations in at least three traversal dimensions.

[0008] Windowing operations are implemented at least partially in hardware, as 2-D windowing operations are implemented in at least one hardware accelerator. The at least one hardware accelerator may include a circuit system configured to perform 2-D windowing operations. The step of assembling partial results may be implemented in software (e.g., by calling a general-purpose programmable circuit system) or in hardware (e.g., by at least one hardware accelerator).

[0009] The steps of assembling partial results may include at least one or any combination of two or more of the following: collecting some or all of the partial results, combining some or all of the partial results, and concatenating some or all of the partial results. Concatenating partial results may include joining them into a single tensor—specifically, joining them into a single tensor with a higher dimension. Combining partial results may include element-wise combination without increasing dimensionality. For example, partial results may be combined by element-wise summation, minimization, maximization, or mean operations. Collecting partial results may include collecting partial results in memory, or collecting partial results and passing them for further processing—for example, collecting partial results and passing them directly to subsequent windowing operations. The manner in which partial results are assembled typically depends on the windowing operation being performed. In particular, it may depend on the type of windowing operation and the size of the window.

[0010] The data includes multiple elements across at least three traversal dimensions. In addition to traversal dimensions, other dimensions such as batch and channel may also exist. The data can also be referred to as a data tensor. When the windowing operation is a convolution, the kernel may include multiple weights (coefficients) across at least three dimensions. The kernel may include multiple filters, each operating independently on the data.

[0011] The at least three traversal dimensions may include, for example, two spatial dimensions, and optionally three spatial dimensions. In one example, the at least three traversal dimensions may include two spatial dimensions and one temporal dimension, or consist of two spatial dimensions and one temporal dimension. Other examples are also possible. Other dimensions may be traversed, including but not limited to one or more frequency dimensions, such as spatial frequency or temporal frequency.

[0012] Windowing operations are limited by a window, which can have a size of at least one or at least two in each of the three traversal dimensions. The data can have a size of at least one, at least two, or at least three in each of the three traversal dimensions. (All permutations and combinations of these sizes are disclosed.)

[0013] Here, a slice refers to a subset of data (or a window) with a reduced size in at least one dimension, but containing the full range of data (or windows) in other dimensions. For example, a 2-D slice of 3-D data reduces the size of the data in one dimension. A 2-D slice of 4-D data reduces the size of the data in two dimensions. In the method of the present invention, each slice can be a 2-D slice. A tensor can be viewed as an array with multiple dimensions. A “slice” of a tensor is also a tensor containing all elements of the original tensor corresponding to one or more specified indices. For example, if tensor T has three dimensions, then a slice U of T obtained in the second dimension with index 5 can be given as follows: for all a in {1...A} and all c in {1...C}, U[a,c] = T[a,5,c], where A is the size of the first dimension and C is the size of the third dimension. In this case, U is called a 2-D slice of the 3-D tensor T. In general, slices can be obtained in any one of the one or more dimensions.

[0014] Typically, slices do not extend at all in the reduced dimension. In other words, the slice has a size of 1 in that dimension. In this case, a 2-D slice contains the full range of data in two dimensions (the two dimensions that are shifted in each 2-D windowing operation) and does not extend in the other dimensions shifted by the windowing operation.

[0015] When replacing multiple component 2-D windowing operations with windowing operations in three or more traversal dimensions, the multiple component 2-D windowing operations for each shift of the window in the remaining dimensions (i.e., at least three traversal dimensions other than the two dimensions of the 2-D windowing operation) can include two or more 2-D windowing operations. The assembly partial result can then include, for each shift of the window in the remaining dimensions, a combination of the partial results of two or more 2-D windowing operations, or a combination thereof.

[0016] Generally speaking, the mathematical operators associated with windowing operations are associative, meaning the result does not depend on how the data is grouped together to perform the windowing operation, and commutative, meaning the order in which the operations are performed can be changed. In the case of convolution or mean pooling, the associated mathematical operator is addition; for example, in the case of max pooling or min pooling, the associated mathematical operators are max or min operators.

[0017] At least one hardware accelerator can be a fixed-function hardware accelerator. In this context, "fixed-function" refers to a characteristic of hardware that the logic implemented by the hardware after manufacturing cannot be reconfigured (or at least cannot be extensively reconfigured). This contrasts with, for example, reconfigurable field-programmable logic. At least one hardware accelerator can be included in an application-specific integrated circuit (ASIC). The behavior of fixed-function hardware can be programmable.

[0018] Windowing operations in at least three traversal dimensions can be defined by a computation graph, and the mapping can include redrawing the computation graph to expand / spread the windowing operations into multiple 2-D windowing operations.

[0019] In each 2D windowing operation, the complete 2D windowing operation can be executed in a single pass on a hardware accelerator configured to perform 2D windowing operations. In this context, a "pass" needs to include a unit of work containing one or more operations for the hardware accelerator to execute. The operations that can be executed together in a single pass depend on the architecture of the hardware accelerator. For example, Figure 10 Hardware accelerators in this system can process a series of operations, such as 2D convolution, bias addition, and pooling, in a single pass. A single pass may require multiple hardware instructions and multiple memory accesses to complete.

[0020] The method may include: selecting two dimensions of at least three traversal dimensions; and mapping the windowing operation to a 2-D windowing operation consisting of multiple components in the selected two dimensions.

[0021] This could include, for example, selecting the two dimensions with the largest window shifts. This would tend to minimize the burden of assembly steps (because the number of partial results and assembly operations would be lower compared to other choices of dimensions) and minimize the number of hardware passes required for 2-D windowing operations.

[0022] The method may further include: when performing one of a plurality of 2-D windowing operations in at least one hardware accelerator, storing at least a portion of a slice of data or at least a portion of a slice of a window in the local memory of at least one hardware accelerator; and subsequently reusing the stored portion when performing another of the plurality of 2-D windowing operations in at least one hardware accelerator.

[0023] When the windowing operation is a convolution operation, the slice of the window can be a slice of the convolution kernel.

[0024] Windowing operations can be convolution operations, where each operation in a 2-D windowing operation is a 2-D convolution operation, and where the assembly partial results include combining partial results by summing the partial results. Assembly units can be configured to combine partial results.

[0025] In some examples, windowing can be a transposed convolution (sometimes referred to as "deconvolution" or strided convolution in neural network literature). Such operations can be used, for example, for upsampling data. In other examples, windowing can be a grouped convolution, where groups of channels in the input data tensor are processed individually.

[0026] Windowing operations can be maximum operations, where each operation in a 2-D windowing operation is a maximum operation, and where the assembly unit is configured to combine partial results by identifying the maximum partial result in the partial results.

[0027] Windowing operations can be minimal operations, where each operation in a 2-D windowing operation is a minimal operation, and where the assembly unit is configured to combine partial results by identifying the minimum partial results in the partial results.

[0028] Windowing operations can be mean pooling operations, where each operation in a 2-D windowing operation is a mean pooling operation, and where assembly units are configured to combine partial results by calculating the mean of partial results.

[0029] Windowing operations may include bias addition operations, wherein a transformation unit is configured to map bias addition to a selected operation in a 2-D windowing operation, wherein the bias addition is performed in at least one hardware accelerator as part of the selected 2-D windowing operation.

[0030] Assembly can be done at least in part by a hardware accelerator component configured to perform element-wise operations on tensors.

[0031] Windowing operations can be performed as part of a neural network comprising multiple layers, the method comprising: identifying windowing operations in at least three traversal dimensions in the multiple layers, wherein at least one hardware accelerator does not support these windowing operations; mapping the neural network to a reconstructed neural network; and implementing the reconstructed neural network by at least one hardware accelerator, wherein the identified windowing operations in the neural network are replaced in the reconstructed neural network by a plurality of 2-D windowing operations that produce partial results and a plurality of element-wise operations that combine the partial results.

[0032] In neural networks, windowing operations can be used for any of a variety of useful tasks involving 3-D (or higher-dimensional) data. When the input data includes video data (comprising two spatial dimensions and one temporal dimension), the task can include analytical or classification tasks such as spatial-temporal segmentation, object recognition, gesture or action recognition, emotion recognition, or video event classification; or tasks such as video compression. When the input data includes volumetric data (comprising three spatial dimensions), the task can include analytical or classification tasks such as 3-D segmentation or 3-D object recognition; or tasks such as 3-D volumetric data compression. If the input data includes 4-D data, comprising three spatial dimensions and one temporal dimension, the task can include analytical or classification tasks such as 3-D spatial-temporal segmentation, 3-D object / action / gesture / emotion recognition, or 3-D event classification; or 4-D data compression. Windowing operations can also be used to restore, denoise, enhance, frame interpolate, or super-resolution data in three or more dimensions, including but not limited to video data and 4-D data comprising three spatial dimensions and one temporal dimension. Data with two spatial dimensions is not limited to natural image data—it can include depth / range data such as that generated by radar, lidar, or acoustic sensors. 3D data can be formed by adding a temporal dimension to such data. Any of the tasks described above as being performed on video data can be efficiently performed on a 3D dataset of temporally varying range images.

[0033] A data processing system for performing windowing operations in at least three traversal dimensions is also provided.

[0034] The data may include one of the following or its derivatives: video data comprising two spatial dimensions and one temporal dimension; and volumetric data comprising three spatial dimensions.

[0035] The data processing system can be implemented in hardware on an integrated circuit. In some implementations, the data processing system can be part of a neural network accelerator (NNA). In other implementations, the NNA can be formed as part of the data processing system.

[0036] The assembly unit can be implemented in at least one hardware accelerator. In particular, the at least one hardware accelerator may include an element-wise operation module configured to combine partial results. The assembly unit may be configured to concatenate partial results by writing them into a memory arranged as a single tensor.

[0037] Alternatively, in some other examples, the assembly unit can be implemented in a general-purpose programmable circuit system. For example, a general-purpose programmable circuit system may include a CPU.

[0038] At least one hardware accelerator may include any two or more of the following or any combination thereof: one or more convolution engines including circuitry configured to perform convolution computations; pooling units including circuitry configured to perform pooling operations; element-wise operation modules including circuitry configured to perform element-wise operations on elements of two or more tensors; and activation units including circuitry configured to implement activation functions for neural network layers.

[0039] Each of these components may include a separate dedicated circuit system. In some embodiments, at least one hardware accelerator may include, for example, an NNA or be composed of it. At least one hardware accelerator may be a fixed-function hardware accelerator, as outlined above.

[0040] The neural network may include a layer comprising a first type of windowing operation in at least three traversal dimensions, and another layer comprising a second type of windowing operation in at least three traversal dimensions, wherein a transformation unit is configured to map the neural network to a reconstructed neural network, wherein the first windowing operation is mapped to a first plurality of 2-D windowing operations, and the second windowing operation is mapped to a second plurality of 2-D windowing operations. At least one hardware accelerator may be further configured to implement the second plurality of 2-D windowing operations. The transformation unit may be configured to select two dimensions of at least two traversal dimensions and map the windowing operation to a plurality of 2-D windowing operations in the selected two dimensions.

[0041] Each operation in the second plurality of 2-D windowed operations can produce a corresponding second part result, wherein an assembly unit is configured to assemble the second part results to produce the result of the second windowed operation. At least one hardware accelerator may include element-wise operation modules configured to appropriately combine some or all of the second part results.

[0042] The transformation unit can be configured to identify, when mapping the neural network to the reconstructed neural network, that the first and second windowing operations are not supported by at least one hardware accelerator, and in response, map them respectively to a first plurality of and a second plurality of 2-D windowing operations. The transformation unit can be configured to map each of the first and second windowing operations such that each operation in the reconstructed neural network is replaced by a plurality of 2-D windowing operations that produce partial results and a plurality of element-wise operations that combine the partial results.

[0043] The data processing system may also include a zero-padding optimization unit, and the data may include zero-padding data, wherein the zero-padding optimization unit is configured to exclude from multiple constituent 2-D windowing operations 2-D windowing operations that would be applied to slices of zero-padding data consisting only of zeros. The zero-padding optimization unit may be part of a transform unit.

[0044] The data processing system may also include a memory, wherein the data processing system is configured to store at least a portion of a slice of data or at least a portion of a slice of a window in the memory when one of a plurality of 2-D windowing operations is performed in at least one hardware accelerator, and to reuse the stored portion when one of a plurality of 2-D windowing operations is subsequently performed in at least one hardware accelerator.

[0045] Therefore, this portion is stored during a 2D windowing operation and reused in subsequent 2D windowing operations. The stored portion can be retrieved from memory without needing to be retrieved from external memory. This helps reduce the bandwidth required for memory access to external memory. In many hardware architectures, operations read from external memory can be costly in terms of time and / or power consumption. Therefore, it is advantageous to reduce the number of read operations from external memory whenever possible.

[0046] The memory can be local memory—for example, implemented on the same semiconductor die as the hardware accelerator, conversion unit, and assembly unit.

[0047] In particular, it may be advantageous to store (cache) a portion of the data if the window's stride in at least one of the remaining dimensions (besides the two dimensions of the 2D windowing operation) is smaller than the window's size in those remaining dimensions. In this case, different slices of the window will be applied to the same data elements in different 2D windowing operations.

[0048] At least one hardware accelerator may include multiple convolution engines, each configured to perform convolution computations, and each convolution engine including multiple elements of multiplication logic and multiple elements of addition logic; and multiple accumulators, each accumulator being configured to perform accumulation computations on the convolution results output by the convolution engines, wherein the convolution engines and accumulators are configured to perform 2-D convolution operations.

[0049] A neural network accelerator, referred to below as NNA, is also disclosed. The NNA is configured to perform the methods outlined above or the methods of various aspects of the present invention.

[0050] A method for manufacturing a data processing system or an NNA as described above, or an aspect of the present invention, using an integrated circuit manufacturing system is also disclosed.

[0051] A method for manufacturing a data processing system or NNA as outlined above using an integrated circuit manufacturing system is also provided, the method comprising: processing a computer-readable description of the data processing system or NNA using a layout processing system to generate a circuit layout description of an integrated circuit embodying the data processing system or NNA; and manufacturing the data processing system or NNA according to the circuit layout description using an integrated circuit manufacturing system.

[0052] A computer-readable code is also provided, configured to cause the methods outlined above or aspects of the present invention to be performed when the code is executed. A computer-readable storage medium is also provided on which the computer-readable code is encoded.

[0053] Computer program code for performing any of the methods described herein may be provided. A non-transitory computer-readable storage medium may be provided having computer-readable instructions stored thereon, which, when executed at a computer system, cause the computer system to perform any of the methods described herein.

[0054] An integrated circuit definition dataset is also provided, which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a data processing system or NNA as outlined above or in various aspects of the present invention.

[0055] Additionally, a non-transitory computer-readable storage medium is provided, on which a computer-readable description of a data processing system or NNA as outlined above or various aspects of the present invention is stored, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the data processing system or NNA.

[0056] A non-transitory computer-readable storage medium is also provided, on which a computer-readable description of a data processing system or NNA as outlined above or in various aspects of the present invention is stored, the computer-readable description causing the integrated circuit manufacturing system to perform the following operations when processed in an integrated circuit manufacturing system: processing the computer-readable description of the data processing system or NNA using a layout processing system to generate a circuit layout description of an integrated circuit embodying the data processing system or NNA; and manufacturing the data processing system or NNA according to the circuit layout description using an integrated circuit manufacturing system.

[0057] An integrated circuit manufacturing system is also provided, which is configured to manufacture a data processing system or NNA as outlined above or in various aspects of the present invention.

[0058] An integrated circuit manufacturing system is also provided, comprising: a non-transitory computer-readable storage medium storing thereon a computer-readable description of a data processing system or NNA configured to perform windowing operations in at least three dimensions, the windowing operations including applying a window having at least three dimensions to data having at least three dimensions, wherein the window is shifted in all three dimensions; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the data processing system or NNA; and an integrated circuit generation system configured to manufacture the data processing system or NNA according to the circuit layout description, wherein the data processing system or NNA includes: a transformation unit configured to map the windowing operations to a plurality of composed 2-D windowing operations, each 2-D windowing operation including applying a slice of the window to a slice of data, wherein the slice of the window is shifted in only two dimensions; at least one hardware accelerator including a circuit system configured to perform a plurality of 2-D windowing operations, each 2-D windowing operation producing a corresponding partial result; and an assembly unit configured to assemble the partial results to produce the result of the windowing operations.

[0059] The layout processing system can be configured to determine the location information of the logic components of a circuit derived from an integrated circuit description in order to generate a circuit layout description of an integrated circuit that embodies a data processing system or NNA.

[0060] As will be apparent to those skilled in the art, the above features can be appropriately combined, and can be combined with any aspect of the examples described herein. Attached Figure Description

[0061] The example will now be described in detail with reference to the accompanying drawings, in which:

[0062] Figure 1 This demonstrates a 2D convolution operation;

[0063] Figure 2 The 2D convolution operation with multiple output channels is shown;

[0064] Figure 3A and Figure 3B The example shows a 3D convolution decomposed into multiple 2D convolutions;

[0065] Figure 4 A computational graph is shown, according to another example, for implementing 3-D convolution as multiple 2-D convolutions;

[0066] Figure 5A The calculation diagram including bias addition is shown;

[0067] Figure 5B A computational graph for implementing 3D convolution transpose is shown;

[0068] Figure 6A This is a simplified block diagram of the data processing system based on the first example;

[0069] Figure 6B This is a simplified block diagram of the data processing system based on the second example;

[0070] Figure 6C This is a simplified block diagram of the data processing system based on the third example;

[0071] Figure 7A This is a flowchart illustrating a method for performing 3D windowing operations, which can be performed by... Figure 6A or Figure 6B The data processing system executes;

[0072] Figure 7B This is a flowchart illustrating a method for implementing a neural network, which can be constructed by... Figure 6C The data processing system executes;

[0073] Figure 8 This is a flowchart illustrating a method for implementing 3D windowing operations, based on another example;

[0074] Figure 9 This illustrates the process of storing and reusing a portion of the kernel or data;

[0075] Figure 10 This is a block diagram of a hardware accelerator for implementing a convolutional neural network, based on an example.

[0076] Figure 11 It shows Figure 10 The example shown is a convolutional engine;

[0077] Figure 12 A computer system in which a data processing system is implemented is shown; and

[0078] Figure 13 A manufacturing system for generating integrated circuits that embody a data processing system is shown.

[0079] The accompanying drawings illustrate various examples. Those skilled in the art will understand that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the drawings represent one example of a boundary. In some examples, it may be that one element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the drawings to indicate similar features. Detailed Implementation

[0080] The following description is given by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art.

[0081] As used herein, a “windowing operation” refers to an operation that applies a function to different positions in a multidimensional input data tensor based on a sliding window that steps across some or all dimensions (referred to simply as a “window traversal dimension” or “traversal dimension”). Windowing operations produce an output data tensor corresponding to the input data tensor. The input and output data tensors may include dimensions other than those traversed by the window (e.g., batch dimensions and / or channel dimensions). A windowing operation can be referred to by the number of dimensions traversed in the operation—for example, an “ND” convolution is a convolution with N traversal dimensions. It should be understood that terms such as “traversal,” “stride,” “slide,” and “shift” do not impose any specific restrictions on the order of data processing. For example, a windowing operation can be performed by sequentially executing computations implied by successive increments of window shifts, but this is not required. In general, computations can be performed in any order, including, if the relevant hardware allows, performing at least some computations in parallel with each other. Implementation schemes will now be described only by way of examples. Most examples will focus on the case of multidimensional convolutions. However, it should be understood that the scope of this disclosure is not limited to convolution as a windowing operation.

[0082] 2D convolution is well-known and widely implemented. It forms the basis of Convolutional Neural Networks (CNNs), which can be used to process and analyze data that is essentially two-dimensional characters. One of the most widespread applications of CNNs for 2D data is image processing and analysis, including image understanding, object detection, and segmentation.

[0083] Figure 1 The figure shows the height H and width W including C. 输入 The input data (input tensor) of the input channel undergoes a 2D convolution operation of 10⁸. Figure 1 In the example shown, C 输入 =3. For example, these channels can represent the three color channels of a color image. Alternatively, in the intermediate layers of a neural network, the channels can represent the outputs of different filters that have been processed in the previous layer. Input data 108 with height K H and width K W Convolution is performed on kernel 120. Figure 1 In-matrix convolutions are indicated by an asterisk (*). The kernel has the same number of channels C as the input data. 输入 2D convolution operations calculate the following weighted sums:

[0084]

[0085] Here, the weight w is the coefficient of the kernel. The data value is represented by x. This calculation is equivalent to shifting (stepping) the kernel across the data in both the height and width dimensions, and for each shift, calculating the inner product of the kernel values, where the data value lies within the window overlaid by the kernel. The height and width dimensions are referred to as being "traversed" by the kernel. The size of each shift in each dimension is called the stride. Figure 1 In the example shown, for simplicity, it is assumed that the stride of each dimension is 1. And for simplicity, there is no padding in this example. The data has H=4 and W=4, and the kernel has K. H =3 and K W =3; therefore, the size of the output data 130 is 2×2.

[0086] In some examples, convolutions can be performed with a stride greater than 1 in one or two dimensions. For example, with a stride of 2, the kernel shifts by two data elements (e.g., pixels) in each step. This reduces the number of inner product calculations to be performed and correspondingly reduces the size of the output. In some examples, the input data can be padded—e.g., with zeros. Padding allows the kernel to be centered on data elements located at or near the boundaries of the input data, where the kernel extends beyond the boundaries. Padding provides additional data values ​​that can be operated on by the relevant elements of the kernel in these cases. This avoids reducing the size of the output tensor compared to the input tensor. Without padding, the kernel can only shift until the boundaries of the kernel reach the boundaries of the data tensor. With appropriate padding and a stride of 1, for example, the size of the output in the two dimensions traversed by the kernel can be the same as the size of the input in those dimensions. Convolutions can also be dilated (also called “holes”), meaning that in addition to using consecutive input data elements on each application of the kernel, the sample locations can be spread across a larger input region. The use of padding and strides other than 1 in 2D convolutions, as well as dilation (and other 2D windowing operations), will be well known to those skilled in the art. Therefore, they will be omitted in much of the following description without loss of generality. It should be understood that the disclosure also applies to windowing operations with any stride and any dilation factor in any traversal dimension when appropriate padding is used.

[0087] Figure 2 Expanding the input data 109 by convolving it with a kernel comprising two filters 122a and 122b Figure 1 Examples. Each filter 122a, 122b and Figure 1 The kernels 120 in the series have the same size and dimensions. The number of filters determines the output channel C. 输出 The quantity. For example... Figure 2 As shown, C输出 =2. The kernel is three-dimensional, as shown in the figure, and has dimensions [C]. 输出 C 输入 K H K W The input tensor 109 is still three-dimensional, with dimensions [C, H, W]. The convolution calculation with the added channel dimension can be written as:

[0088]

[0089] Similarly, this is simplified by ignoring stride and padding without loss of generality. The size of the output data 132 is 2×2×2 (that is, the number, height, and width of the output channels are all equal to 2). Again, the height and width are referred to as the "traversal" dimensions.

[0090] Additional dimensions, called batch dimensions, can also exist. Figure 2 The input tensor 109 is not shown in the diagram. In this case, the input tensor 109 is four-dimensional, with dimensions [B, C, H, W]. The convolution computation with added batch dimensions can be written as:

[0091]

[0092] It should be noted that although the kernel and input tensor are four-dimensional, this is still a 2-D convolution because the kernel only shifts in the height and width directions.

[0093] The general concept of convolution can be extended to additional traversal dimensions. For example, a new traversal dimension labeled "D" (for "depth") can be added. The input tensor then increases from a 4-D tensor to a 5-D tensor: [B, C, D, H, W], with a size of D in the depth dimension. Similarly, the kernel increases to a 5-D kernel: [C output, C input, KD, KH, KW], with a size of KD in the depth dimension. An additional summation is introduced into the convolution computation, now including shifts across the three traversal dimensions [D, H, W]. This 3-D convolution computation can be written as:

[0094]

[0095] For clarity and simplicity, stride and padding are again omitted in this simplified formula without loss of generality. Summation across four dimensions is computationally intensive, with each step performed in each of the three dimensions. Therefore, efficient implementation is desired.

[0096] Other windowing operations commonly implemented in neural networks include pooling operations such as min-pooling, max-pooling, and average (or mean) pooling. While traditionally applied as 2-D operations (with two traversal dimensions), networks can also contain pooling operations with three traversal dimensions. It is also desirable to implement these 3-D operations efficiently. For example, a 2-D max-pooling operation on a 4-D input tensor [B, C, H, W] can be described by the following equation:

[0097]

[0098] Similar to convolution, this can be extended to 3D max pooling operations on 5-D input tensors [B, C, D, H, W], described by the following equation:

[0099]

[0100] In 2D max pooling, there are two traversal dimensions (height and width), while in 3D max pooling, there are three traversal dimensions (depth, height, and width). Equations can be written for min pooling and mean pooling operations that traverse three dimensions. It's important to note that there are no weights in max or min pooling—therefore, in the classical sense, there is no "kernel." However, in the context of this invention, these are still windowing operations, meaning they apply a multi-dimensional window to the input data tensor, and for this shift of the window, only the data values ​​falling within that window are considered to calculate the corresponding output value.

[0101] Another windowing operation sometimes implemented in neural networks is transposed convolution, also known as convolution transpose, deconvolution, or fractionally-strided convolution. The 2D convolution transpose applied to a 4D input tensor [B, C, H, W] can be described by the following equation:

[0102]

[0103] This can be extended to the 3-D convolution transpose applied to a 5-D input tensor [B, C, D, H, W], described by the following equation:

[0104]

[0105] Neural networks exist as fixed-function hardware accelerators known as "neural network accelerators" (NNAs). In this context, "fixed-function" refers to the characteristic of hardware that the logic implemented by the hardware after manufacturing cannot be reconfigured. This contrasts with, for example, reconfigurable field-programmable logic. The behavior of fixed-function hardware can be programmable. For example, a fixed-function hardware accelerator can be an application-specific integrated circuit (ASIC). NNAs typically include dedicated, optimized hardware for fast and efficient computation of 2-D operations, such as convolution and pooling. The inventors have recognized that existing NNA hardware specifically designed for handling 2-D windowing operations, including, for example, convolution and pooling operations, can be used to efficiently implement 3-D windowing operations. Multiple 2-D convolutions can be constructed and combined to implement 3-D convolutions. The following equation shows how a 3-D convolution operation can be constructed such that one dimension of the traversed dimensions (the depth dimension in this example) includes the outermost loop (i.e., the outermost summation). The inner part of the computation (within the square brackets) consists of multiple 2-D convolution operations, as referenced above. Figure 2 The operations that are introduced.

[0106]

[0107] Performing this reconstruction along a dimension to produce a combination of 2-D operations can be referred to differently as “expanding,” “ungrouping,” or “factoring out” the dimension. Although factoring out was applied to the depth dimension in the example above, it should be understood that this method is generally applicable: any one of the three traversal dimensions can be selected and factored out, thus leaving the other two traversal dimensions as multiple 2-D operations within the internal part of the computation.

[0108] Similarly, the inventors have recognized that other 3D windowing operations can also be efficiently implemented by constructing the computation into multiple 2D operations and implementing these 2D operations using existing NNA hardware. This applies, for example, to pooling as discussed above. 3D max-pooling operations can be computed as multiple 2D max-pooling operations as described below:

[0109]

[0110] In this example, degrouping is also performed along the depth dimension, but the same technique can be applied to the height or width dimension. It should be understood that a 3-D minimum pooling (or 3-D mean pooling) operation can be similarly computed as a combination of multiple 2-D minimum pooling operations (or 2-D mean pooling operations, respectively).

[0111] Similarly, the above 3D convolution transpose operation can be constructed into multiple 2D convolution transpose operations, as shown below (again, ungrouping along the depth dimension without restriction or loss of generality):

[0112]

[0113] As can be deduced from the aforementioned equation, when a 3-D windowing operation is mapped to multiple constituent 2-D windowing operations, each operation in the 2-D windowing operation produces a partial result. These partial results need to be assembled to produce the result of the 3-D windowing operation. The appropriate way to assemble the partial results depends on the operations being performed. In the case of convolution, the relevant operation is addition; therefore, the partial results are combined by summing them along the ungrouped dimension. In the special case where the convolution kernel has a size of 1 in the ungrouped dimension (i.e., in the example above, when K...), ... D When the summation is 1, it is not necessary to sum, and only partial results are collected or concatenated at the output of the 2-D windowed operation. For example, this would apply to convolutions with a 1×1×1 kernel. In the case of max pooling, the relevant operation is the max operator; therefore, they will be assembled by combining partial results using the max operator. Similarly, for min pooling, they will be assembled by combining partial results using the min operator. In the convolution transpose operation, partial results can be assembled by summation and optional concatenation. In certain special cases (depending on the stride), the partial results of the 3-D convolution transpose can be assembled entirely by collection—optionally, by concatenating them appropriately in the ungrouped dimensions, thus gathering them into a tensor. As previously stated, it should be understood that factors can be de-applied to any traversal dimension except the depth dimension.

[0114] Neural network accelerators may include dedicated hardware for performing element-wise operations on tensors. Here, "element-wise" refers to binary operations. Make

[0115]

[0116] This includes element-wise summation, max-sum multiplication, and so on. Hardware for element-wise operations can be used to efficiently combine partial results. For example, element-wise addition can be used to combine partial results from multiple 2D convolution operations. Element-wise max can be used to combine partial results from multiple 2D max-pooling operations; and element-wise min can be used to combine partial results from multiple 2D min-pooling operations.

[0117] Figures 3A to 3B It shows that Figure 2The 2D convolution is extended to three dimensions and the 3D convolution operation is implemented as a set of 2D convolution examples. Figure 3A and Figure 3B The input tensor 110 is shown extended in the depth dimension. In this example, it has a size D = 3 in that depth dimension. The data 110 is shown as three "slices" 1101 to 1103, each slice having a size of 1 in the depth dimension. Each of these slices is equivalent to... Figure 2 The example uses input data 109. (Example:) Figure 2 As shown, the batch dimension has been suppressed. Therefore, the input tensor is a 5-D tensor, [B, C, D, H, W]. The meaning of each dimension in the convolution depends on the application. In some examples, the three traversal dimensions can be spatial dimensions [D, H, W]. In other examples, one or more traversal dimensions do not need to be spatial dimensions. For example, a traversal dimension can be a temporal dimension, a temporal frequency dimension, or a spatial frequency dimension.

[0118] Similarly, the kernel is a 5-D data structure [C 输出 C 输入 K D K H K W ], where K in this example D =2. Assume the kernel's stride is 1 in each of the three dimensions. The kernel is depicted as two "slices" 1221 to 1222, each with a size of 1 in the depth dimension. Each slice is equivalent to... Figure 2 Kernel 122 in the middle. Figure 3A The kernel is shown in a first position (first shift), where the first slice 1221 of the kernel is aligned with the first slice 1101 of the input tensor, and the second slice 1222 of the kernel is aligned with the second slice 1102 of the input tensor. When presented in this manner, it can be seen that, for the first shift, the computation of the 3D convolution consists of two 2D convolutions and a subsequent element-wise summation. The first slice 1101 of the input tensor is convolved with the first slice 1221 of the kernel to produce a first partial result 1321. The second slice 1102 of the input tensor is convolved with the second slice 1222 of the kernel to produce a second partial result 1322. (Each of these partial results is equivalent to...) Figure 2 The result of the 2-D convolution in the kernel is 132. To produce the output of the 3-D convolution with the kernel shifted in the depth dimension, the two partial results are summed by a summing block 1401.

[0119] Figure 3BThe kernel at its second position (second shift) according to the stride of the convolution in the D dimension is shown, where the first slice 1221 of the kernel is aligned with the second slice 1102 of the input tensor, and the second slice 1222 of the kernel is aligned with the third slice 1103 of the input tensor. Similarly, it can be seen that the computation of the 3-D convolution for the second shift consists of two 2-D convolutions followed by an element-wise summation. The second slice 1102 of the input tensor is convolved with the first slice 1221 of the kernel to produce a third partial result 1323. The third slice 1103 of the input tensor is convolved with the second slice 1222 of the kernel to produce a fourth partial result 1324. To produce the output 1522 of the 3-D convolution for this shift of the kernel in the depth dimension, the two partial results 1323 and 1324 are summed by a summation block 1402. Therefore, the complete output of a 3D convolution (with dimensions 2×2×2×2) can be generated by performing four 2D convolutions and then two element-wise sums of the partial results produced by the 2D convolutions.

[0120] The reconstruction of 3D convolutions over multiple 2D convolutions can also be represented in the form of a computational graph. Neural networks can be conceived as computational graphs; the advantage of this representation is that 3D convolutions in a neural network can be replaced with equivalent computational subgraphs, making deployment on NNAs that support 2D convolutions and element-wise summation straightforward. Figure 4 An example is shown below. Here, the original input tensor 510 has a size of D = 5 in the depth dimension. The input tensor 510 is shown as five slices 5101 to 5105, each slice having a size of 1 in the depth dimension. It is desired to correlate each of these slices with a size of K in the depth dimension. D A kernel of size 3 is used for convolution. In this example, the stride in the depth dimension is equal to 2. Based on these parameters, the output 550 will have a size of 2 in the depth dimension. The output 550 consists of two slices 5501 to 5502. The first slice 5501 can be produced by performing three 2-D convolutions 5601 to 5603, where the first three slices 5101 to 5103 of the input tensor are convolved with the three slices of the kernel, respectively. The outputs of these 2-D convolutions are summed element-wise at summing block 5401 to produce the first slice 5501 of the output tensor.

[0121] Similarly, the second slice 5502 can be generated by performing three 2D convolutions 5604 to 5606, where the last three slices 5103 to 5105 of the input tensor are convolved with the three slices of the kernel, respectively. The outputs of these 2D convolutions are summed element-wise at summing block 5402 to produce the second slice 5502 of the output tensor. Note that the weights in 2D convolution 5601 are the same as the weights in 2D convolution 5604. In both cases, these are the weights from the first slice of the kernel. Similarly, the weights in 2D convolution 5602 are the same as the weights in 2D convolution 5605 (the second slice of the kernel); and the weights in 2D convolution 5603 are the same as the weights in 2D convolution 5606 (the third slice of the kernel). It should be understood that in this way, the processing of the depth dimension of the original 3D convolutions has been reproduced in the topology of the computation graph.

[0122] Figure 5A Another example is shown, where Figure 4 The example is extended to include bias addition. Bias addition is a commonly used feature in CNNs, and it is desirable to be able to implement bias addition efficiently in the context of 3D convolutions. NNAs that support effective 2D convolutions typically support bias addition as part of a 2D convolution. The inventors of this invention have recognized that bias addition in 3D convolutions can be included in one of the 2D convolutions that produce a partial result of the 3D convolution. This can be made to implement bias addition efficiently by incorporating it into a single pass of the 2D convolution function on the NNA. Bias addition can be incorporated into any 2D convolution that begins to produce a single output slice of the 3D convolution. This can be done in... Figure 5A As seen in the computational graph, bias summation 5701 is included at the output of convolution 5601, and bias summation 5702 is similarly incorporated into the output of convolution 5604. In both cases, the convolution weights and biases are the same.

[0123] For other 3D windowing operations, such as pooling operations, it is possible to draw with... Figures 3A to 4 The diagram is a fairly schematic representation. For example, if 3D convolution is replaced by 3D max pooling, then... Figures 3A to 4 The 2D convolution will be replaced by 2D max pooling operations, and Figures 3A to 4 The element-wise addition operation will be replaced by the element-wise maximum operation.

[0124] Figure 5BAnother example of the computation curve is shown, this time for 3D convolution transpose. As explained above, 3D convolution transpose is constructed as multiple 2D convolution transpose operations. The original input tensor 511 has a size of D = 3 in the depth dimension. This original input tensor is shown as three slices 5111 to 5113, each slice having a size of 1 in the depth dimension. The kernel has a size of K in the depth dimension. D =3. In this example, the stride in the depth dimension is 2. Based on these parameters, the output 551 will have a size of 7 in the depth dimension. Output 551 consists of seven slices 5511 to 5517. The first slice 5511 and the second slice 5512 are produced by performing 2D deconvolutions 5611 to 5612 on the first slice 5111 of the input tensor, respectively. The third slice 5513 is produced by performing two 2D deconvolutions 5613 to 5614 on the first slice and the second slice 5111 to 5112 of the input tensor, respectively, and summing the results element-wise at the summing block 5411. The fourth slice 5514 is produced by performing a 2D deconvolution 5615 on the second slice 5112 of the input tensor. The fifth slice 5515 is produced by performing two 2D deconvolutions 5616 and 5617 on the second and third slices 5112 and 5113 of the input tensor, respectively, and summing the results element-wise at summation box 5412. The sixth slice 5516 and the seventh slice 5517 are produced by performing 2D deconvolutions 5618 and 5619 on the third slice 5113 of the input tensor, respectively. It should be understood that the same weights are used for deconvolutions 5611, 5614, and 5617 (all of which use the first slice of the kernel); deconvolutions 5612, 5615, and 5618 (all of which use the second slice of the kernel); and deconvolutions 5613, 5616, and 5619 (all of which use the third slice of the kernel).

[0125] Figures 6A to 6C Three variations of the data processing system based on the example are shown. Figures 7A to 7B A flowchart of the associated methods that can be implemented by a data processing system is shown.

[0126] Figure 6A It is a block diagram of the data processing system 600 based on the first example. Figure 7AA flowchart illustrating a method performed by a data processing system 600 according to this example is shown. The data processing system includes a transformation unit 610; a hardware accelerator 620; an assembly unit 630; and memory, specifically local memory 640. The data processing system may also include an optional zero-padding optimization unit 650 (shown in dashed outline), which may be included within the transformation unit 610. The transformation unit 610 receives the definition of a 3-D windowing operation to be performed by the data processing system 600 and the definition of a data tensor. The definition of the data tensor includes at least the size of the data in each dimension. In this example, the 3-D windowing operation is a 3-D convolution. In step 710, the transformation unit 610 selects two dimensions from the three traversal dimensions of the windowing operation. The transformation unit maps the 3-D convolutions to a set of 2-D convolutions in these two selected dimensions, as referenced above. Figure 4 as well as Figure 5A and Figure 5B The partial results from the 2D convolutions are assembled (particularly combined) along the remaining third dimension not selected in step 710. In this example of the invention, the transformation unit 610 selects the two largest dimensions; therefore, the 3D convolutions are unfolded into 2D convolutions along the smallest dimension, and the combination of partial results is also unfolded along the smallest dimension. In this way, the transformation unit 610 arranges the 3D convolutions into a relatively small number of relatively large 2D convolutions. (However, in other examples, the dimensions may be selected according to different rules.)

[0127] In step 720, the transformation unit 610 performs a mapping. The result of this mapping defines a set of slices of the input data and a set of slices of the kernel. These slices extend in the selected dimensions and do not extend in the unselected third dimension; that is, each slice in the unselected third dimension has a size of 1. (Refer to the above text) Figures 3A to 5B The method described decomposes 3-D convolutions into a set of 2-D convolutions between data slices and kernel slices.

[0128] In this example, the data is not zero-padded, and no zero-padded optimization unit is used. Slices of the input data are stored in the local memory 640 of the data processing system while the corresponding 2D convolutions are performed. Similarly, when performing 2D convolution operations, the coefficients (weights) representing the kernel slices are stored in the local memory 640.

[0129] In step 730, the circuitry of the hardware accelerator 620 performs a set of 2D convolution operations. That is, as described above... Figures 3A to 5BAs explained, hardware accelerator 620 convolves appropriate slices of the input tensor with appropriate slices of the kernel. Depending on the size of the local memory, it may not be possible to fit the entire input tensor and kernel simultaneously in memory 640. Therefore, data can be loaded (stored) into local memory as needed for use in any given 2D convolution.

[0130] The output from each 2D convolution is a partial result. These partial results are provided to the assembly unit 630 by the hardware accelerator 620. In step 740, the assembly unit 630 assembles (in particular combines) the partial results from each 2D convolution to produce the final result of the 3D convolution. Since the 3D windowing operation is a 3D convolution in this example, the assembly unit combines the partial results by summing them. The result of the summation is a slice of the output (e.g., as referenced above). Figure 4 and Figure 5A These slices are collected to provide the output of a 3D convolution. In some cases, the collected slices can be passed to the next layer of the neural network separately from each other. Alternatively, in some cases, the slices can be concatenated into a single output tensor by stacking the slices along the unfolded dimension. This can be done by writing the slices to appropriate locations in local memory 640 or external memory—for example, to occupy a single contiguous block of memory.

[0131] There are special cases where summation is not required, and only partial results from 2-D convolutions are collected and optionally concatenated. This occurs when the kernel has a size of 1 in the ungrouped dimension. For example, this happens when implementing a 3-D convolution with a 1×1×1 kernel. The assembly unit 630 is configured to assemble partial results in any appropriate manner based on the type and window size of the 3-D windowing operation.

[0132] exist Figure 6A In the example, the assembly unit 630 is shown separately from the hardware accelerator 620. In some implementations, this may indeed be the case. For example, the assembly unit may be implemented in a general-purpose programmable circuit system such as a CPU. The CPU can be programmed to perform necessary assembly operations on partial results of a 2-D convolution. Therefore, in Figure 6A In the example, the transformation unit 610 instructs the hardware accelerator 620 to perform multiple 2-D convolution operations, and instructs the assembly unit 630 to sum, collect, and optionally concatenate the resulting partial results as needed to produce the original 3-D convolution result.

[0133] In some cases, integrating the functionality of the "assembly unit" into a single, unified part of the hardware accelerator may be advantageous. This is in... Figure 6BThe diagram shown is a block diagram of a data processing system 600' according to a second example. Data processing system 600' differs from data processing system 600 because the assembly unit 620' is part of a hardware accelerator 630'. If the hardware accelerator 630' is implemented in a neural network accelerator, at least some of the functions of the assembly unit can be performed by a dedicated circuit system specifically for element-wise operations. In the example of the invention, the 3-D windowing operation is convolution, and the element-wise operation is addition. In data processing system 600', the transformation unit 610' provides the hardware accelerator 620' with a complete set of hardware-implementable operations to perform the 3-D windowing operation. In other words, the transformation unit 610' instructs the hardware accelerator 620' to perform multiple 2-D windowing operations and instructs how to assemble the results of these 2-D operations (including appropriate element-wise summation in the example of 3-D convolution). As a result of the 3-D convolution, the result of the element-wise summation is collected and output. Apart from these modifications, the method implemented by data processing system 600' is similar to... Figure 7A The methods shown are essentially the same.

[0134] One or more 3D windowing operations can be performed as part of a neural network comprising multiple layers. In this case, it may be advantageous for the transform unit to process the entire neural network rather than attempting to map individual 3D windowing operations to corresponding multiple 2D operations. This is in... Figure 6C The diagram shows a block diagram of a data processing system 600 according to a third example. The method performed according to this example... Figure 7B As shown in the flowchart. System 600 is similar to Figure 6B The system 600', because assembly unit 630" is part of hardware accelerator 620". Figure 6B The difference between system 600' and system 605' is that the transformation unit 610" receives the complete definition of a neural network as input, which may include one or more 3-D windowing operations. The definition of the neural network also includes the definition of the data tensors processed at each layer of the network. In step 705, the transformation unit 610" identifies any 3-D windowing operations in the neural network that are not supported by the hardware accelerator 620". Step 710" is... Figure 7AAn extension of step 710—For each of the identified 3-D windowing operations, transformation unit 610 selects the two dimensions that will be traversed when mapping the 3-D operation to the 2-D operation. Then, in step 720, transformation unit 610 maps the entire neural network to the reconstructed neural network. This includes mapping each of the identified 3-D windowing operations to a corresponding plurality of 2-D windowing operations, and defining how to assemble the partial results of these 2-D windowing operations. The 2-D windowing operations and the assembly operations are suitable for implementation by hardware accelerator 620 (i.e., supported); therefore, the reconstructed neural network is referred to as “hardware implementable”. Layers of the original neural network that do not include unsupported 3-D windowing operations can be preserved as invariant in the reconstructed neural network. In step 735, the hardware accelerator 620” implements the reconstructed neural network. This includes implementing a corresponding plurality of constituent 2-D windowing operations for each of the identified 3-D windowing operations, and assembling the results of these 2-D windowing operations. The assembly is performed using the assembly unit 630”, and the assembly is performed again within the hardware accelerator 620”.

[0135] In this example, zero-padding optimization can also be performed. Generally, zero-padding may need to be performed at several layers within the neural network; therefore, zero-padding will be performed not only on the input data tensors but also on the tensors representing the outputs of intermediate layers. This can be accomplished by a hardware accelerator 620” operating under the control of the transformation unit 610”.

[0136] Figure 8 The extension performed by data processing systems 600 and 600' is shown. Figure 7A The flowchart of the method. It should be understood that the process executed by data processing system 600”... Figure 7B The method can be extended similarly. According to... Figure 8 The method will include optional bias addition. Bias addition is typically combined with convolution. It can also be done in... Figure 8 Optional zero-padding optimization is performed in the method. The steps associated with optional zero-padding optimization are described in... Figure 8 It is shown in the dashed outline.

[0137] Typically, zero-padding is performed on tensors in neural networks so that the output of a layer has the same dimension as its input (depending on the stride). This principle also applies to higher-dimensional windowing operations, such as the 3D convolution operations illustrated in this invention. Figure 8In the example, assume the data is zero-padded before being stored in local memory 640. This includes zero-padding at the edges of the data in each traversal dimension. When the data is subsequently structured as slices, this would mean that there are slices consisting entirely of zeros. Any convolution with an array of zeros will produce zeros as a result because convolution involves multiplying the input by weights (coefficients). These zeros have no effect on the final result because partial results are combined by addition (and adding zeros to anything has no effect). Based on this understanding, in embodiments of the invention, these unnecessary convolution operations will be omitted.

[0138] Step 710 and Figure 7A The corresponding steps are the same. In step 720a, the transformation unit 610 maps the 3-D convolution operation to a set of 2-D convolution operations (in the selected two dimensions), this time including as referenced above. Figure 5A The bias addition operation is described above. In step 722, the zero-padding optimization unit 650 identifies 2D convolutions that will be completely zero-based due to zero-padding. In step 724, the zero-padding optimization unit 650 excludes the identified unnecessary 2D convolutions from the operations to be performed by the hardware accelerator. In other words, these operations are not performed when the 2D operations are performed by the hardware accelerator 620 in step 730a. Omitting unnecessary operations saves computation time, memory, bandwidth, and power.

[0139] Since slices consisting entirely of zero-padding are not actually used for 2D windowing operations, it may not be necessary to store them in local memory 640. This allows for more efficient use of memory capacity by omitting redundant data and eliminating redundant operations.

[0140] exist Figure 8 In the example, it is assumed that the data is already zero-padded when it is stored in local memory 640. This is not necessary. In other examples, the data can be stored in local memory 640 without padding, and the zero-padding can be added on the fly by hardware accelerator 620 as multiple 2-D windowing operations are being performed (steps 730 / 730a). In this case, hardware accelerator 620 also does not need to instantiate a slice of data consisting entirely of zero-padding. In other words, hardware accelerator 620 can add zero-padding only in the two dimensions selected in step 710 without adding it in the unselected dimensions.

[0141] Figure 9 Step 730 is shown in more detail. In particular, Figure 9 The process shown can be used to improve the efficiency of memory bandwidth usage when the hardware accelerator 620 is performing 2D convolution operations. This can be applied to, for example... Figure 7A Step 730 and as shown Figure 8 The steps shown in step 730, and Figure 7B Step 735 in the text. The basis for this enhancement is the recognition that data and kernel weights are reused to some extent when this set of 2-D convolutions is implemented. This is in Figure 4 and Figure 5A This is most easily seen in the examples. As mentioned earlier, in these examples, each slice of the kernel is used for two separate 2D convolutions. The weights in 2D convolution 5601 are the same as the weights in 2D convolution 5604. In both cases, these are the weights from the first slice of the kernel. Similarly, the weights in 2D convolution 5602 are the same as the weights in 2D convolution 5605 (the second slice of the kernel); and the weights in 2D convolution 5603 are the same as the weights in 2D convolution 5606 (the third slice of the kernel). It can also be seen that the third slice 5103 of the input tensor is used twice in 2D convolutions 5603 and 5604.

[0142] In the original implementation, this reuse pattern would be ignored—the relevant slices of data and kernels would be loaded from external memory and stored in local memory 640 each time they were needed for 2-D convolution. However, according to an example of the invention, data or weights, or both, can be stored once in local memory 640 and used at least twice for two different 2-D convolutions. In effect, this allows for caching of data and / or weights across different convolutions in multiple 2-D convolutions. This is reflected in steps 732 and 734 performed by the data processing system when implementing 2-D convolutions in hardware accelerator 620. In step 732, the system stores slices (at least a portion) of the kernels or slices (at least a portion) of the data in local memory 640 for later reuse in subsequent 2-D convolution operations. In step 734, hardware accelerator 620 reuses the stored slices in subsequent 2-D convolution operations. Therefore, for example, the weights used for convolution 5601 (i.e., the first slice of the kernel) can be stored in local memory and held there until the hardware accelerator 620 reuses the weights to perform convolution 5604. Alternatively or additionally, at least a portion of the third slice of the input tensor 5103 can be stored in local memory 640 and held there after convolution 5603 is completed. When convolution 5604 is performed, this (at least partial) slice of the input tensor can then be reused without further reading from external memory. The amount of data stored and reused may depend on the capacity of local memory 640.

[0143] Transform unit 610 can be configured to sort the group of 2-D convolution operations to increase the reuse of data and coefficients stored in local memory 640. For example, transform unit 610 can instruct hardware accelerator 620 to perform convolution 5605 immediately after convolution 5602, as this would allow the stored weights to be reused between these consecutive convolutions.

[0144] The examples above have focused on 3D convolution operations as windowing operations. However, this disclosure is not limited to 3D windowing operations or convolution operations. Any windowing operation with associative and commutative properties can be decomposed / unfolded into multiple equivalent 2D windowing operations, similar to the 3D convolution discussed above. It should be understood that the step of assembling partial results should be selected based on the windowing operation being performed. Assembly may include combining partial results (as in step 740) and / or collecting or concatenating partial results. When performing a convolution operation, combination 740 is based on element-wise summation. For a 3D max pooling operation, combination will be based on the element-wise max operator (i.e., returning the largest partial result among the partial results). Similarly, for a 3D min pooling operation, combination will be based on the element-wise min operator. Other windowing operations can be performed based on those operations already mentioned. For example, mean pooling can be implemented using a convolution operation.

[0145] In the example described above, the original windowing operation to be performed is a 3-D windowing operation, which is expanded into multiple 2-D windowing operations in one dimension. The same strategy can be applied to windowing operations in higher dimensions. For example, a 4-D windowing operation can be expanded into multiple 2-D windowing operations in two dimensions. In this way, windowing operations in any number of dimensions can be simplified into a set of 2-D windowing operations, which makes it possible to efficiently implement 2-D windowing operations using hardware accelerators.

[0146] Therefore, data processing systems 600, 600', and 600" can be used to efficiently perform 3-D (and higher-dimensional) operations as well as traditional 2-D operations. Furthermore, systems optimized for 2-D windowing operations can be used to efficiently perform 3-D and higher-dimensional windowing operations with minimal or no modification. It should be understood that when dimensions are listed in the foregoing description (e.g., a 4-D tensor is represented as "[B, C, H, W]"), the order of the dimensions generally does not have special significance. In different implementations, dimensions may be ordered differently within the tensor.

[0147] Hardware accelerators suitable for efficiently implementing 2D windowing operations (especially 2D convolution operations) will now be described.

[0148] Figure 10A hardware accelerator 200 configured to implement a CNN is shown. The hardware accelerator includes a digital logic circuit system configured to receive data (including weights and input tensors) and commands for processing the data. The hardware accelerator 200 includes a memory interface 210, an input buffer controller 215, a command decoder 220, a coefficient buffer controller 225, a coefficient buffer 230, an n-input buffer 235, an n-convolution engine 240, an n-accumulator 245, an activation module 255, a normalization module 265, a shared buffer 270, a pooling module 275, and an element-wise operation module 285. According to the example, the hardware accelerator 200 can be used to implement 2D operations and assemble partial results. In particular, it is suitable for... Figure 6B or Figure 6C The hardware accelerator 620' or 620" is provided. In this case, Figure 6B and Figure 6C The "local memory" 640 is composed of Figure 10 The coefficient buffer 230, input buffer 235, accumulation buffer 250 and shared buffer 270 are provided together.

[0149] Memory interface 210 is configured to provide an interface between hardware accelerator 200 and external memory (not shown). The external memory can be considered a separate module from hardware accelerator 200. Command or configuration data may include, for example, information about weights and data size and format, as well as their location in the external memory.

[0150] Memory interface 210 is configured to receive from external memory weights and data to be used in computation within the CNN, as well as command information for controlling the operations of hardware accelerator 200. The received weights (also referred to herein as coefficients) are passed to coefficient buffer controller 225, and the received data is passed to input buffer controller 215. The received commands are passed to command decoder 220, which is then configured to decode the commands and subsequently publish control information to elements of the hardware accelerator, including coefficient control buffer controller 225 and input buffer controller 215, to control how weights and input data are stored in buffers.

[0151] Weights and input data received from external memory via memory interface 210 during a read from external memory may form only a portion of the weights and input data for a single layer, all of which may be used to process that single layer, or may include weights and input data for processing multiple layers. For example, weights received from external memory may form the weights for a single layer, and the received input data may form only a portion of the input data for a single layer (or vice versa). Any combination of data and weights across one or more layers may be received from external memory in a single read from memory (e.g., using a burst read).

[0152] In practice, the amount of weights and data received in a single read from external memory will depend on the size of coefficient buffer 230 and input buffer 235. Weights are passed from coefficient buffer controller 225 to coefficient buffer 230, and received data is passed from input buffer controller 215 to multiple input buffers 235a to 235n. The number of input buffers will depend on the specific implementation of accelerator 200, but can take any value. Input data is shared across all input buffers 235a to 235n. Each input buffer forms an effective bank, allowing the number of input buffers to be increased or decreased depending on the application.

[0153] Input buffers 235a to 235n are connected to each of the multiplexers because each convolutional engine 240a to 240n needs access to all valid "repositories" of the input data. Each multiplexer is configured to select an output from one of the input buffers and pass the value output from the selected input buffer to the corresponding convolutional engine 240a to 240n. Additionally, weights from coefficient buffer 230 are provided as a second input to each convolutional engine 240a to 240n. The convolutional engines are configured to perform convolution computations on the received input data using the weights received from coefficient buffer 230. The resulting outputs of each convolutional engine 240a to 240n are provided as inputs to the corresponding accumulators of the multiple accumulators 245a to 245n. Figure 11 An exemplary implementation of a convolution engine 240n is shown, which includes multiple elements of multiplication logic 310 and multiple elements of addition logic 320.

[0154] Each accumulator 245a to 245n is connected to an accumulation buffer 250. The accumulation buffer 250 is configured to store the accumulated result received from each accumulator 245a to 245n. The accumulation buffer 250 is connected to a memory interface 210. Therefore, the accumulation buffer 250 is configured to send data to and receive data from external memory via the memory interface 210. Specifically, the accumulation buffer 250 is configured to store and retrieve its value via the memory interface 210, as will be described in more detail below. The accumulation buffer 250 is connected to the inputs of the accumulators 245a to 245n and is configured to feed values ​​back to the accumulators 245a to 245n to enable accumulation calculations.

[0155] Accumulation buffer 250 is configured to pass accumulated values ​​to activation module 255. Activation module 255 is configured to execute at least one of a plurality of different activation functions. In some embodiments, activation module 255 may be configured to operate as a rectified linear unit (ReLU).

[0156] The value calculated by activation module 255 is passed to normalization module 265 for processing via shared buffer 270. Normalization module 265 is configured to execute one of many different normalization functions. Local response normalization layer functions can be executed within the same plane of the input data. Local response normalization layer functions can also be executed across planes.

[0157] Normalization module 265 can be configured to implement one or more of the normalization functions described above. Specifically, normalization module 265 can be configurable to execute any one of the three functions described above and any other suitable function. In this way, normalization module 265 can receive signaling to indicate which function the normalization module should execute. This can be done using one of a number of predefined functions, such that the control signaling received by the normalization function indicates which predefined functions the normalization module 265 should execute.

[0158] The results stored in the shared buffer 270 are passed to the memory interface 210, which can store the results in external memory or pass the results back to the input buffer for further processing, without having to pass them to external memory first.

[0159] The shared buffer 270 is configured for two purposes: buffering values ​​from the activation module 255 until the correct sample is available for normalization and / or buffering values ​​from the normalization unit until the correct sample is available for pooling. In this way, the shared buffer is used to achieve storage efficiency, since in many implementations normalization and pooling are not combined in the same process.

[0160] The element-wise operation module 285 includes a circuit system configured to perform element-wise operations on tensors received from the accumulation buffer 250 and / or the activation module 255. Supported element-wise operations include element-wise addition, multiplication, and maximum (or minimum) values ​​of the corresponding elements of the tensor.

[0161] Although Figure 10 The hardware accelerator illustrates a specific order of module arrangement and data processing in the hardware implementation, but it should be understood that the order of specific computations and cross-layer data processing required can vary.

[0162] In some layers, functions executed by the activation 255, normalization 265, pooling 275, and element-wise 285 modules can be executed. In other layers, only one of these functions can be executed, and not necessarily in the order listed in the hardware accelerator 200. To achieve a configurable order of processing these functions, each of the activation 255, normalization 265, pooling 275, and element-wise 285 modules can be configured to receive control signaling that configures the module to bypass mode, in which no function is executed and the input values ​​simply pass through the module without change.

[0163] In some layers, data for a specific layer may first be processed by convolutional engines 240a-n, and then by activation, normalization, pooling, and element-wise modules 255, 265, 275, and 285. In these layers, the output from convolutional engines 240a-n is passed to accumulation buffer 250 via accumulators 245a-n, and then to activation, normalization, pooling, and element-wise modules 255, 265, 275, and 285 for further processing. In other layers, data may need to be processed differently. For example, data may first be processed by activation, normalization, pooling, and element-wise modules 255, 265, 275, and 285, and then by convolutional engines 240a-n.

[0164] In these arrangements, data can be directly passed to the activation module via the accumulation buffer 250, which has already received input data directly from the memory interface 210, which has received data from external memory. In this way, the processing performed by the convolution engines 240a-n and accumulators 245a-n is effectively skipped, and the data can be directly passed to the activation 255, normalization 265, pooling 275, and element-wise 285 modules. Then, once the processing using the activation, normalization, pooling, and element-wise modules 255, 265, 275, and 285 is complete, the resulting value can be passed to the input buffer controller 215 via the memory interface 210. In some arrangements, the resulting value can first be passed to external memory via the memory interface 210 and then retrieved from external memory before use.

[0165] In other arrangements, memory interface 210 can pass the obtained value to input buffer controller 215 instead of passing the value to external memory. By avoiding the need to pass the calculated values ​​using activation, normalization, pooling, and element-wise modules 255, 265, 275, 285 to external memory, memory bandwidth is reduced, and therefore, latency in processing data is also reduced.

[0166] By providing the aforementioned features, the hardware accelerator 200 can potentially handle various configurations of CNNs by providing the ability to perform memory-to-memory computations. These memory-to-memory computations do not need to be directly associated with any particular layer. In other words, the hardware implementation 200 is adapted to the specific CNN to be implemented. Advantageously, due to the linear placement of activation, normalization, pooling, and element-wise modules 255, 265, 275, 285, it is possible to execute these functions back-to-back without having to retrieve data from external memory.

[0167] In some implementations, the connection order of the activation, normalization, pooling, and element-wise modules 255, 265, 275, 285 can be varied. For example, the activation, normalization, and pooling modules 255, 265, 275 can be connected in reverse order, such that the pooling module is connected to the accumulation buffer 250, and the activation module is connected to the memory interface 210.

[0168] As explained above, the hardware accelerator 200 can be used in multiple passes to efficiently implement multiple composed 2-D windowing operations, thereby composing windowing operations in a large number of dimensions. The hardware accelerator 200 can also be used to assemble (e.g., combine) partial results of the composed 2-D windowing operations. In this context, the convolution engine 240 and accumulator 245 can be used to implement 2-D convolution; the pooling module 275 can be used to implement 2-D max pooling, min pooling, or mean pooling. The element-wise operation module 285 can be used to combine partial results from these 2-D operations.

[0169] For example, if the original 3-D operation is a mean pooling operation, then this can be mapped by transformation unit 610' or 610" to a plurality of 2-D mean pooling operations performed by pooling module 275. These are then combined by calculating the mean of the partial results produced by these 2-D mean pooling operations. This can be done by element-wise operation module 285, where element-wise addition and element-wise multiplication by a scalar are performed to implement the mean.

[0170] Figure 12A computer system is shown in which the data processing system described herein can be implemented. The computer system includes a CPU 902, an NNA 904, a memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 919. A processing block 910 (corresponding to data processing systems 600, 600', 600") is implemented on the NNA 904. In other examples, the processing block 910 may be implemented on the CPU 902. Components of the computer system may communicate with each other via a communication bus 920. The memory 906 may provide the previously mentioned "external memory".

[0171] Figures 6A to 6C The data processing systems 600, 600', and 600" are shown as comprising numerous functional blocks. This is merely illustrative and not intended to define a strict division between the different logical elements of such entities. Each functional block may be provided in any suitable manner. It should be understood that the intermediate values ​​described herein formed by the data processing systems do not need to be physically generated by the data processing systems at any point in time, and may merely represent logical values ​​that conveniently describe the processing performed by the data processing systems between their inputs and outputs.

[0172] The data processing system described herein can be embodied in hardware on an integrated circuit. The data processing system described herein can be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., a fixed logic circuit system), or any combination thereof. The terms “module,” “function,” “component,” “element,” “cell,” “block,” and “logic” are used herein to generally denote software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, cell, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein can be executed by one or more processors that execute code that causes the processor to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other techniques to store instructions or other data and can be accessed by a machine.

[0173] As used herein, the terms computer program code and computer-readable instructions refer to any kind of executable code that is to be executed by a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code defining integrated circuits (e.g., hardware description languages ​​or netlists), and code expressed in languages ​​such as C++. or Executable code is code expressed in programming languages ​​such as [programming languages ​​not specified in the original text]. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes the processor of a computer system that supports the executable code to perform the tasks specified by that code.

[0174] A processor, computer, or computer system can be any kind of device, machine, or special-purpose circuit, or a collection or part thereof, that has the processing power to execute instructions. A processor can be any kind of general-purpose or special-purpose processor, such as a CPU, GPU, NNA, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0175] This invention also intends to cover software defining the configuration of hardware as described herein, such as HDL (Hardware Description Language) software, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium on which computer-readable program code in the form of an integrated circuit definition dataset is encoded may be provided, which, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a data processing system or NNA configured to perform any of the methods described herein, or to manufacture a data processing system or NNA including any of the devices described herein. The integrated circuit definition dataset may, for example, be an integrated circuit description.

[0176] Therefore, a method for manufacturing a data processing system or NNA as described herein can be provided at an integrated circuit manufacturing system. Furthermore, an integrated circuit definition dataset can be provided, which, when processed in the integrated circuit manufacturing system, enables the method for manufacturing the data processing system or NNA to be executed.

[0177] Integrated circuit definition datasets can be in the form of computer code, such as netlists, code for configuring programmable chips, or hardware description languages ​​suitable for manufacturing at any level in integrated circuits, including register-transfer level (RTL) code, high-level circuit representations (such as Verilog or VHDL), and low-level circuit representations (such as OASIS(RTM) and GDSII). Higher-level representations (such as RTL) that logically define hardware suitable for manufacturing in integrated circuits can be processed on a computer system configured to generate manufacturing definitions of integrated circuits within the context of a software environment that includes definitions of circuit elements and rules for combining these elements to generate the manufacturing definitions of the integrated circuits defined by that representation. As is typically the case where software executes at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate the manufacturing definitions of the integrated circuits, executing code that defines the integrated circuits to generate the manufacturing definitions of said integrated circuits.

[0178] Now refer to Figure 13 This describes an example of processing integrated circuit definition datasets at an integrated circuit manufacturing system in order to configure the system as a data processing system for manufacturing, based on an example.

[0179] Figure 13 An example of an integrated circuit (IC) manufacturing system 1002 is shown, configured to manufacture a data processing system as described in any of the examples herein. Specifically, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g., defining a data processing system as described in any of the examples herein), process the IC definition dataset, and generate an IC (e.g., embodying the data processing system as described in any of the examples herein) based on the IC definition dataset. Through the processing of the IC definition dataset, the IC manufacturing system 1002 is configured to manufacture integrated circuits embodying the data processing system as described in any of the examples herein.

[0180] The layout processing system 1004 is configured to receive and process an IC definition dataset to determine a circuit layout. Methods for determining a circuit layout based on an IC definition dataset are known in the art and may involve, for example, synthesizing RTL code to determine the gate-level representation of the circuit to be generated, for example, in relation to logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined based on the gate-level representation of the circuit. This can be done automatically or with user intervention to optimize the circuit layout. Once the layout processing system 1004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0181] As is known in the art, IC generation system 1006 generates ICs according to a circuit layout definition. For example, IC generation system 1006 may implement a semiconductor device manufacturing process for generating ICs, which may involve a multi-step sequence of photolithography and chemical processing steps, during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask, which can be used in the photolithography process to generate ICs according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of computer-readable code, which IC generation system 1006 can use to form a suitable mask for generating ICs.

[0182] The various processes performed by the IC manufacturing system 1002 can all be implemented in one location, for example, by one party. Alternatively, the IC manufacturing system 1002 can be a distributed system, allowing some processes to be performed at different locations and by different parties. For example, some of the following stages can be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition dataset to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0183] In some implementations, when the integrated circuit manufacturing definition dataset is processed in the integrated circuit manufacturing system, the integrated circuit manufacturing system can generate devices as described herein. For example, the integrated circuit manufacturing definition dataset, as referred to above... Figure 13 The configuration of the integrated circuit manufacturing system described herein can produce devices as described in this document.

[0184] In some examples, an integrated circuit definition dataset may include software running on hardware defined at the dataset, or software running in combination with hardware defined at the dataset. Figure 13In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined at the integrated circuit definition dataset during the manufacturing of the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0185] Compared to known implementations, the concepts set forth in this application can lead to performance improvements in devices, apparatuses, modules, and / or systems (and in the methods implemented herein) when implemented. Performance improvements may include one or more of increased computational performance, reduced latency, increased throughput, and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules, and systems (e.g., in integrated circuits), trade-offs can be made between performance improvements and physical implementation methods, thereby improving manufacturing processes. For example, a trade-off can be made between performance improvements and layout area to match the performance of known implementations but using less silicon. This can be accomplished, for example, by reusing functional blocks serially or sharing functional blocks among elements of a device, apparatus, module, and / or system. Conversely, the concepts set forth in this application that lead to improvements in the physical implementation of devices, apparatuses, modules, and systems (such as reduced silicon area) can be traded off for performance improvements. This can be accomplished, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0186] The applicant has independently disclosed each individual feature described herein, as well as any combination of two or more such features, to the extent that such features or combinations can be implemented based on the specification as a whole, in accordance with the common knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein. In view of the foregoing description, those skilled in the art will understand that various modifications can be made within the scope of this invention.

Claims

1. A method for performing a windowing operation in at least three traversal dimensions, the windowing operation comprising applying a window having at least three dimensions to data having at least three traversal dimensions, wherein the window is shifted in all three traversal dimensions, the method comprising: Choose two of the at least three traversal dimensions described in (710); The windowing operation is mapped (720) to a plurality of 2-D windowing operations in two selected dimensions, each 2-D windowing operation comprising applying a slice of the window to a slice of the data, wherein the slice of the window is shifted only in two dimensions; Each of the plurality of 2-D windowing operations (730) is performed by at least one hardware accelerator, each 2-D windowing operation producing a corresponding partial result; and The partial results of assembly (740) are used to produce the results of the windowing operation. The two selected dimensions are the two dimensions with the largest window shifts among the at least three traversal dimensions.

2. The method of claim 1, wherein the data includes zero-padding data, and wherein mapping the windowing operation to the plurality of constituent 2-D windowing operations comprises: Exclude (724) from the 2-D windowing operation of the multiple components that will be applied to the 2-D windowing operation of the slice of zero-filled data consisting only of zeros.

3. The method of any one of claims 1 to 2, further comprising, when performing one of the plurality of constituent 2-D windowing operations in the at least one hardware accelerator, storing (732) at least a portion of the slice of the data or at least a portion of the slice of the window in the local memory of the at least one hardware accelerator, and When one of the multiple 2-D windowing operations is subsequently performed in the at least one hardware accelerator, the portion stored in (734) is reused.

4. The method of any one of claims 1 to 2, wherein the windowing operation is one of the following operations: Convolution operation, wherein each operation in the 2-D windowing operation is a 2-D convolution operation, and wherein compiling the partial results includes combining them by summing the partial results; Maximum operation, wherein each of the 2-D windowed operations is a maximum operation, and wherein assembling the partial results includes combining them by identifying the largest partial results among the partial results; Minimal operations, wherein each of the 2-D windowed operations is a minimal operation, and wherein assembling the partial results includes combining them by identifying the minimum partial results among the partial results; and Mean pooling operation, wherein each operation in the 2-D windowing operation is a mean pooling operation, and wherein combining the partial results includes combining them by calculating the mean of the partial results.

5. The method of any one of claims 1 to 2, wherein the windowing operation includes bias addition, wherein mapping the windowing operation (720a) to the plurality of constituent 2-D windowing operations includes mapping the bias addition to a selected operation in the 2-D windowing operation, wherein the bias addition is performed in the at least one hardware accelerator as part of the selected 2-D windowing operation (730a).

6. The method of any one of claims 1 to 2, wherein the data includes one of the following or its derivative: Video data, the video data including two spatial dimensions and one temporal dimension; and Volume data, which includes three spatial dimensions.

7. The method of any one of claims 1 to 2, wherein the windowing operation is performed as part of a neural network comprising multiple layers. The method includes: In the plurality of layers, windowing operations are identified (705) in at least three traversal dimensions, wherein the at least one hardware accelerator does not support the windowing operations; Map the neural network to the reconstructed neural network (720''); as well as The reconstructed neural network is implemented by the at least one hardware accelerator (735). The identified windowing operation in the neural network is replaced in the reconstructed neural network by a plurality of 2-D windowing operations that produce partial results and a plurality of element-wise operations that combine the partial results.

8. A data processing system (600) for performing windowing operations in at least three traversal dimensions, the windowing operations comprising: The data processing system comprises applying a window with at least three dimensions to data having at least three traversal dimensions, wherein the window is shifted across all three traversal dimensions. The transformation unit (610) is configured to map (720) the windowing operation to a plurality of 2-D windowing operations, each 2-D windowing operation including applying a slice of the window to a slice of the data, wherein the slice of the window is shifted in only two dimensions; At least one hardware accelerator (620) includes a circuit system configured to perform the plurality of 2-D windowing operations described in (730), each 2-D windowing operation producing a corresponding partial result; and An assembly unit (630) is configured to assemble (740) the partial results to produce the result of the windowing operation. The transformation unit (610) is configured as follows: Select two dimensions from the at least three traversal dimensions (710), wherein the selected two dimensions are the two dimensions with the largest window shift among the at least three traversal dimensions; and The windowing operation is mapped (720) to a 2-D windowing operation consisting of multiple components in the two selected dimensions.

9. The data processing system of claim 8, wherein the at least one hardware accelerator comprises any one or any combination of the following: One or more convolution engines, including circuitry configured to perform convolution computations; A pooling unit includes a circuit system configured to perform pooling operations; An element-wise operation module includes a circuit system configured to perform element-wise operations on the elements of two or more tensors; The assembly unit; as well as An activation unit includes a circuit system configured to implement activation functions for neural network layers.

10. The data processing system of any one of claims 8 to 9, wherein the windowing operation in at least three traversal dimensions is a first windowing operation, forming a layer of a neural network comprising multiple layers, the plurality of 2D windowing operations being a first plurality of 2D windowing operations, and the partial result being a first partial result. The neural network includes another layer that includes a second windowing operation in at least three traversal dimensions. The transformation unit is configured to map the neural network to the reconstructed neural network, wherein the first windowing operation is mapped to a first plurality of 2-D windowing operations, and the second windowing operation is mapped to a second plurality of 2-D windowing operations, wherein at least one hardware accelerator is further configured to implement the second plurality of 2-D windowing operations; and optionally, each operation in the second plurality of 2-D windowing operations produces a corresponding second partial result, wherein the assembly unit is configured to assemble the second partial results to produce the result of the second windowing operation.

11. The data processing system as described in claim 10, wherein: (A) The transformation unit is configured to, when mapping the neural network to the reconstructed neural network, The at least one hardware accelerator, identified by symbol (705), does not support the first windowing operation and the second windowing operation, and In response, they are mapped (720'') to the first plurality of 2-D windowing operations and the second plurality of 2-D windowing operations, respectively. And / or (B) The first windowing operation and the second windowing operation are arranged in successive layers of the neural network, and wherein each first partial result forms a data slice of a corresponding windowing operation input into the second plurality of windowing operations.

12. A method for manufacturing a data processing system as described in any one of claims 8 to 11 using an integrated circuit manufacturing system, the method comprising: A layout processing system is used to process a computer-readable description of the data processing system in order to generate a circuit layout description of an integrated circuit that embodies the graphics processing system. as well as The data processing system is manufactured using an integrated circuit manufacturing system based on the circuit layout description.

13. A computer-readable storage medium having computer-readable code encoded thereon, the computer-readable code being configured to cause the method of any one of claims 1 to 7 to be performed when the code is executed.

14. A non-transitory computer-readable storage medium storing thereon a computer-readable description of a data processing system as claimed in any one of claims 8 to 11, wherein, when processed in an integrated circuit manufacturing system, the computer-readable description causes the integrated circuit manufacturing system to perform the following operations: A layout processing system is used to process the computer-readable description of the data processing system to generate a circuit layout description embodying the integrated circuits of the data processing system; and The data processing system is manufactured using an integrated circuit manufacturing system based on the circuit layout description.

15. An integrated circuit manufacturing system, comprising: A non-transitory computer-readable storage medium having stored thereon a computer-readable description of a data processing system as described in any one of claims 8 to 11; A layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the data processing system; as well as An integrated circuit manufacturing system configured to manufacture the data processing system according to the circuit layout description.

Images