Acceleration of 1x1 convolution in convolutional neural networks

Abstract

The present disclosure relates to acceleration of 1x1 convolutions in convolutional neural networks. A convolution accelerator includes a feature row buffer, a kernel buffer, multiply-accumulate clusters, and mode control circuitry. In a first mode of operation, the mode control circuitry stores feature data in the feature row buffer and stores kernel data in the kernel buffer. The data stored in the buffers is transferred to the MAC clusters of the convolution accelerator for processing. In a second mode of operation, the mode control circuitry stores feature data in the kernel buffer and stores kernel data in the feature row buffer. The data stored in the buffers is transferred to the MAC clusters of the convolution accelerator for processing. The second mode of operation can be used to efficiently process 1xN kernels, where N is an integer greater than or equal to 1.

Description

Technical Field

[0001] This disclosure generally relates to convolution accelerators, such as those used in learning / inference machines (e.g., artificial neural networks (ANNs), such as convolutional neural networks (CNNs)). Background Technology

[0002] Various computer vision, speech recognition, and signal processing applications can benefit from the use of learning / inference engines, which can rapidly perform hundreds, thousands, or even millions of concurrent operations. As discussed in this disclosure, learning / inference engines can fall under the technical names of machine learning, artificial intelligence, neural networks, probabilistic inference engines, accelerators, etc. Traditional learning / inference engines can deliver hundreds of teraflops (e.g., one million per second (10^6)). 12 It has the computational power of 100 floating-point operations.

[0003] Such learning / inference engines can include or utilize CNNs, such as Deep Convolutional Neural Networks (DCNNs). A DCNN is a computer-based tool that processes large amounts of data and adaptively "learns" by integrating relevant features from the data proximal to make broad predictions and by improving those predictions based on reliable conclusions and new integrations. DCNNs are arranged in multiple "layers," with different types of predictions performed at each layer. Hardware accelerators, including convolutional accelerators, are typically used to speed up the processing of large amounts of data by DCNNs. Summary of the Invention

[0004] In one embodiment, the convolution accelerator includes a feature row buffer, a kernel buffer separate from the feature row buffer, a multiply-accumulate (MAC) cluster, and mode control circuitry coupled to the feature row buffer, the kernel buffer, and the MAC cluster. In a first operating mode of the convolution accelerator, the mode control circuitry stores feature data in the feature row buffer, stores kernel data in the kernel buffer, transfers feature data from the feature row buffer to the MAC cluster, and transfers kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the convolution accelerator, the mode control circuitry again stores feature data in the kernel buffer, stores kernel data in the feature row buffer, transfers feature data from the kernel buffer to the MAC cluster, and transfers kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1.

[0005] In one embodiment, a system includes: a streaming engine that streams feature and kernel data in operation; and a convolution accelerator coupled to the streaming engine, wherein the convolution accelerator receives streams of feature and kernel data from the streaming engine in operation. The convolution accelerator includes a feature row buffer, a kernel buffer, a multiply-accumulate cluster coupled to the feature row buffer and the kernel buffer, and a mode control circuitry coupled to the feature row buffer, the kernel buffer, and the MAC cluster. In a first operating mode of the convolution accelerator, the mode control circuitry stores feature data in the feature row buffer, stores kernel data in the kernel buffer, transfers feature data from the feature row buffer to the MAC cluster, and transfers kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the convolution accelerator, the mode control circuitry stores feature data in the kernel buffer, stores kernel data in the feature row buffer, transfers feature data from the kernel buffer to the MAC cluster, and transfers kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1.

[0006] In one embodiment, a method includes streaming feature data and kernel data to a convolutional accelerator, and convolving the streamed kernel data with the streamed feature data. The convolution includes, in a first operating mode of the convolutional accelerator, storing feature data in a feature row buffer of the convolutional accelerator, storing kernel data in a kernel buffer of the convolutional accelerator, transferring feature data from the feature row buffer to the MAC cluster of the convolutional accelerator, and transferring kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the convolutional accelerator, the convolution includes storing feature data in a kernel buffer, storing kernel data in a feature row buffer, transferring feature data from the kernel buffer to the MAC cluster, and transferring kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1.

[0007] In one embodiment, the content of a non-transitory computer-readable medium is configured with a convolution accelerator having multiple operating modes to convolve streaming kernel data with streaming feature data. The convolution includes: in a first operating mode of the multiple operating modes of the convolution accelerator, storing feature data in a feature row buffer of the convolution accelerator, storing kernel data in a kernel buffer of the convolution accelerator, transferring feature data from the feature row buffer to the MAC cluster of the convolution accelerator, and transferring kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the multiple operating modes of the convolution accelerator, the convolution includes storing feature data in a kernel buffer, storing kernel data in a feature row buffer, transferring feature data from the kernel buffer to the MAC cluster, and transferring kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1. In one embodiment, the content includes instructions executed by the convolution accelerator. Attached Figure Description

[0008] One or more embodiments will be described below with reference to the accompanying drawings.

[0009] Figure 1 This is a conceptual diagram illustrating the digit recognition task.

[0010] Figure 2 This is a conceptual diagram illustrating an image recognition task.

[0011] Figure 3 This is a conceptual diagram illustrating an example of a CNN.

[0012] Figure 4 This is a conceptual diagram showing an example convolutional layer of a CNN.

[0013] Figure 5 This is a conceptual diagram illustrating the stride of a convolutional layer in a CNN.

[0014] Figure 6 This is a conceptual diagram illustrating the application of padding to the input feature map during convolution to maintain the height and width dimensions.

[0015] Figure 7 This is a conceptual diagram illustrating the batch loading of feature data.

[0016] Figure 8 This is a conceptual diagram illustrating batch convolution processing.

[0017] Figure 9 This is a functional block diagram of an embodiment of an electronic device or system employing a convolution accelerator mode control circuit device.

[0018] Figure 10This is a conceptual diagram illustrating the processing of a convolution operation with a 3×3 kernel, using an embodiment of a convolution accelerator.

[0019] Figure 11 This is a conceptual diagram showing convolutional layers with 3×3 and 1×1 kernels, respectively.

[0020] Figure 12 This is a conceptual diagram illustrating an embodiment of a convolution accelerator that uses a feature row buffer to store feature row data and a kernel buffer to process convolution operations with a 1×1 kernel in an operating mode where the components of the convolution accelerator are turned off.

[0021] Figure 13A and Figure 13B (Hereinafter referred to as Figure 13) is a conceptual diagram illustrating two operating modes of the convolution accelerator according to an embodiment.

[0022] Figure 14 This is a conceptual diagram illustrating an embodiment of a convolution accelerator that uses feature row data stored in a kernel buffer and kernel data stored in a feature row buffer to process convolution operations with a 1×1 kernel in a deep operating mode of a convolution accelerator.

[0023] Figure 15 This is a functional block diagram of an embodiment of a convolution accelerator, which includes a mode control circuit device for controlling the operation of the convolution accelerator in a deep operating mode.

[0024] Figure 16 This is a conceptual diagram illustrating the features and kernel data flow paths in the deep operating mode of a convolutional accelerator according to an embodiment.

[0025] Figure 17 This is a conceptual diagram illustrating the performance of three convolution operations with a kernel value of 1×1 kernels per operating cycle of a convolution accelerator operating in deep operating mode according to an embodiment.

[0026] Figure 18 This is a conceptual diagram illustrating the data path of feature data in a convolutional accelerator operating in a deep operation mode according to an embodiment.

[0027] Figure 19 This is a conceptual diagram illustrating the data path of kernel data in a convolutional accelerator operating in deep operation mode according to an embodiment.

[0028] Figure 20 This is a conceptual diagram illustrating how, according to an embodiment, a convolutional accelerator operating in deep operation mode utilizes three MAC units to share each kernel value and utilizes a pair of MAC clusters to share feature data for each pixel.

[0029] Figure 21This is a conceptual diagram illustrating a convolution operation associated with a 1×2 kernel in a convolution accelerator operating in deep operation mode, according to an embodiment.

[0030] Figure 22A and Figure 22B (Hereinafter referred to as Figure 22) is a conceptual diagram illustrating a convolution operation associated with a 1×3 kernel in a convolution accelerator operating in a deep operating mode, according to an embodiment.

[0031] Figure 23 This is a conceptual diagram illustrating a convolution operation associated with a 1×2 kernel in a convolution accelerator operating in a deep operation mode using an adder tree, according to an embodiment.

[0032] Figure 24 A logic flowchart is shown, generally illustrating an embodiment of a process for controlling a convolution accelerator in multiple operating modes. Detailed Implementation

[0033] To provide a thorough understanding of the various disclosed embodiments, certain specific details are illustrated in the following description, together with the accompanying drawings. However, those skilled in the art will recognize that the disclosed embodiments can be practiced with or without one or more of these specific details, or in various combinations of other methods, components, devices, materials, etc. In other instances, well-known structures or components associated with the environment of this disclosure in a hardware accelerator environment, including but not limited to interfaces, power supplies, physical component layouts, convolution accelerators, multiply-accumulate (MAC) circuits, etc., are not shown or described to avoid unnecessarily obscuring the description of the embodiments. Furthermore, the various embodiments can be methods, systems, devices, computer program products, etc.

[0034] Throughout this specification, claims, and drawings, unless the context otherwise requires, the following terms have the meanings relevant to this document. The term “this document” refers to the specification, claims, and drawings relating to this application. The phrases “in one embodiment,” “in another embodiment,” “in various embodiments,” “in some embodiments,” “in other embodiments,” and their variations refer to one or more features, structures, functions, limitations, or characteristics of the invention, and are not limited to the same or different embodiments unless the context otherwise indicates. As used herein, the term “or” is an inclusive “or” operator and is equivalent to the phrases “A or B, or both” or “A or B or C, or any combination thereof,” and lists with additional elements are treated similarly. The term “based on” is not exclusive and allows for reliance on additional features, functions, aspects, or limitations not described, unless the context otherwise requires. Furthermore, throughout this specification, the meanings of “a,” “an,” and “the” include both singular and plural references.

[0035] CNNs are particularly well-suited for recognition tasks, such as identifying numbers or objects in images, and can provide highly accurate results. Figure 1 This is a concept diagram illustrating the digit recognition task. Figure 2 This is a conceptual diagram illustrating the image recognition task.

[0036] A CNN is a specific type of deep neural network (DNN) that performs convolutions on a three-dimensional feature data tensor (width × height × depth) with one or more layers. The first layer is the input layer, and the last layer is the output layer. Intermediate layers can be called hidden layers. The most commonly used layers are convolutional layers, fully connected or dense layers, and pooling layers (max pooling, average pooling, etc.). The data exchanged between layers is called features or activations. Each layer also has a set of learnable parameters, often called weights or kernels. Figure 3 This is a conceptual diagram illustrating an example of a CNN as AlexNet. The CNN shown has a set of convolutional layers interleaved with max-pooling layers, followed by a set of fully connected or dense layers.

[0037] The parameters of a convolutional layer consist of a set of learnable filters called kernels. Each kernel has three dimensions: height, width, and depth. The height and width are typically constrained to a range (e.g., [1, 11]). The depth typically extends to the entire depth of the input feature data. Each kernel slides across the width and height of the input features and computes a dot product. At the end of processing, a set of two-dimensional feature maps is obtained as the result. In a convolutional layer, many kernels are applied to the input feature maps, each kernel producing a different feature map as the result. The depth of the output feature tensor is also called the number of output channels. Figure 4 This is a conceptual diagram illustrating the application of a kernel to a feature map, producing a two-dimensional feature map with a height of 4 and a width of 4.

[0038] Convolutional layers can also have additional parameters that can be defined for the convolutional layer, instead of the learned parameters. Such parameters are called hyperparameters. For example, a convolutional layer can have hyperparameters that include stride and padding.

[0039] The stride hyperparameter represents the step size used to slide the kernel across the input feature map. Figure 5 This is a conceptual diagram comparing stride 1 and stride 2. When the stride is greater than 1, the output feature map will be smaller than the input feature map.

[0040] The padding hyperparameter indicates the number of zeros to be added along the height, width, or both of the input feature map. Padding parameters can be used to control the size of the output feature map produced by the convolution. Figure 6 This is a conceptual diagram illustrating the application of padding to the input feature map. The padding preserves the input feature dimensions along the height and width of the feature map.

[0041] The feature data of a convolutional layer can have hundreds or even thousands of channels, the number of which corresponds to the depth of the feature data and kernel data. Therefore, the feature and kernel data are often loaded into memory in batches. Figure 7 This is a conceptual diagram illustrating the concept of batch loading feature data. The feature data is divided into batches along the depth dimension, with each batch having the same height, width, and depth. The kernel depth is typically the same as the depth of the input feature map, thus solving similar problems through batch processing.

[0042] As shown in the figure, these batches have a height of 5, a width of 5, and a depth of 4. The batches are typically written to memory sequentially, with the first batch of writes completed before the second batch of writes begins. Figure 7 The arrows in the diagram illustrate an example sequence for writing a batch of data to memory. A similar batching process is typically applied to kernel data, where each batch of kernel data has the same kernel height, kernel width, and depth as the feature data batch. Each batch of feature data is convolved with the kernel data of the relevant batch, and a feedback mechanism is used to accumulate the batch results. Figure 8 The concept diagram illustrates the concept of batch processing of convolution.

[0043] As can be seen, computations performed by CNNs or other neural networks typically involve repetitive computations on large amounts of data. Therefore, computing systems with hardware accelerators can be employed to improve the efficiency of performing operations associated with CNNs.

[0044] Figure 9 This is a functional block diagram of an embodiment of an electronic device or system 100 to which the described embodiments can be applied. System 100 includes one or more processing cores or circuits 102. Processing core 102 may include, for example, one or more processors, state machines, microprocessors, programmable logic circuits, discrete circuits, logic gates, registers, etc., and various combinations thereof. The processing core can control the overall operation of system 100, the execution of application programs by system 100 (e.g., a program that uses CNN to classify images), etc.

[0045] System 100 includes one or more memories 104, such as one or more volatile and / or non-volatile memories, which may store all or part of instructions and data related to, for example, control of system 100, applications and operations performed by system 100. One or more of the memories 104 may include a memory array that may be shared by one or more processes executed by system 100 during operation.

[0046] System 100 may include one or more sensors 160 (e.g., image sensors, audio sensors, accelerometers, pressure sensors, temperature sensors, etc.), one or more interfaces 170 (e.g., wireless communication interfaces, wired communication interfaces, etc.), and other circuitry 180 that may include antennas, power supplies, one or more built-in self-test (BIST) circuits, etc., as well as a main bus system 190. The main bus system 190 may include one or more data, address, power, and / or control buses coupled to the various components of system 100.

[0047] System 100 also includes one or more hardware accelerators 110 that accelerate the performance of one or more operations associated with implementing a CNN during operation. The illustrated hardware accelerator 110 includes one or more convolution accelerators 112 to facilitate efficient execution of convolutions associated with the convolutional layers of a CNN. The convolution accelerator includes: a feature row buffer memory 114 that routinely stores feature data rows during operation; a kernel buffer memory 116 that routinely stores kernel data during operation; and a cluster of one or more multiply-accumulate (MAC) units or circuits 118 that use the buffered kernels and feature data to perform convolution operations during operation. The illustrated hardware accelerator 110 also includes a streaming engine 150 and a streaming switch 155. During operation, the streaming engine 150 sends a data stream. For example, the streaming engine 150 may stream data, such as feature data or kernel data stored in memory 104, to the convolution accelerator 112 via the streaming switch 155.

[0048] Kernel size can vary between CNNs and between convolutional layers within a single CNN. For example, in Figure 3 The diagram illustrates convolutional layers with kernels of height and width dimensions of 11×11, 5×5, and 3×3. Historically, 3×3 kernels have been the most common. Therefore, convolutional accelerators are often designed to efficiently support kernel computation within defined kernel height and width dimensions (e.g., 3×3 kernels).

[0049] For example, feature row buffer 114 may typically be a deep single-port memory that stores several rows of feature data (e.g., three rows) and each row has a large number of storage elements (e.g., 1024 elements per row). Feature row data is typically reused with multiple cores, and a single port is usually sufficient for feature row buffer 114 because computation can usually begin once the feature row buffer is full.

[0050] Kernel buffer memory 116 can typically be a small, wide, and shallow two-port memory (e.g., 96 bits wide by 16 memory locations). Because kernel buffers are wide, they are relatively expensive, and for this reason, they are typically shallow to reduce memory cost (e.g., reduce area cost). Given the shallow depth of the memory, to provide efficient bandwidth, kernel buffer 116 can be a two-port memory to simultaneously read stored kernel data for the current convolution operation and load kernel data into the kernel buffer for subsequent convolution operations.

[0051] Figure 10 It shows the use of, such as Figure 9 A conceptual diagram of the processing of a convolution operation with a 3×3 kernel in an embodiment of the convolution accelerator 112. Feature data is stored in a feature row buffer 114 (e.g., three rows of feature data that can be processed together), and kernel data is stored in a kernel buffer 116 (e.g., supporting kernels of various sizes up to a threshold size, wherein the storage in the buffer is optimized for 3×3 kernel data, for example). Convolution operations are performed using MAC clusters 118 (e.g., 72 MACs organized into 6 clusters 118, each cluster 118 including 12 MACs (three groups, four MAC units 120 per group)) to operate on the data received from the feature row buffer and the kernel buffer. Figure 10 Examples of other typical components of a convolution accelerator (e.g., stream interface 142, stream buffer 144, adder tree 146, configuration register 148, etc.) are shown, without reference to Figure 10 Detailed description. Larger kernel sizes (e.g., 4×4, 11×11, etc.) can often be handled using techniques such as software kernel decomposition.

[0052] However, many CNNs have one or more layers with kernels that have relatively small height and width dimensions. For example, hidden layers or inner layers, especially dense or deep layers, can have a large number of feature data channels and can often use kernels with height and width dimensions less than 3×3, such as 1×1, 1×2, or 1×3, 1×N kernels. In particular, deep or dense convolutional layers with feature data that have a large number of channels convolved with a 1×1 depth kernel are becoming increasingly popular in CNNs. Figure 11 This is a conceptual diagram showing convolutional layers with 3×3 and 1×1 kernels, respectively.

[0053] It is possible to process kernels with smaller height and width dimensions (e.g., 1×1 kernels) by disabling some components of the convolution accelerator. This can be inefficient in many ways, as referenced... Figure 12 The concept diagram discusses, Figure 12An example of processing a convolutional layer with a 1×1 kernel is shown by disabling components of a convolutional accelerator optimized for use with larger kernels. Feature data is stored in feature row buffer 114, and kernel data is stored in kernel buffer 116.

[0054] To avoid congestion due to bandwidth saturation at the output, the number of input channels to the MAC unit (limited by the depth of kernel data that can be stored in the kernel buffer) is maintained to be greater than or equal to the number of output channels of the MAC unit. For a kernel with a height and width of 1×1, a typical kernel buffer depth limitation can limit the number of input channels to the MAC cluster 118 to, for example, 24 channels. Therefore, in the example considered, the number of output channels of the MAC cluster will be limited to 24 channels due to the bottleneck created by the size of the kernel buffer memory to avoid bandwidth saturation at the output.

[0055] In the example configuration of 72 MAC units organized into 6 clusters C0-C5, each containing 12 MAC120 units discussed above, only 4 MAC units 120 of the 12 MAC units in each cluster can be activated, the remaining MAC units remain idle, and each kernel processes only one output pixel per cycle. As shown in the figure, kernels 1-4 are provided to the first cluster of 12 MAC units, in which only 4 of the 12 MAC units are used; kernels 5-8 are provided to the second cluster of 12 MAC units, in which only 4 of the 12 MAC units in the second cluster are used, and so on.

[0056] Furthermore, compared to processing with 3×3 kernels using the same or similar convolutional accelerator configuration, the ratio between the extracted kernel data and the number of operations performed using that extracted kernel data drops dramatically. Therefore, when employing a solution that disables the convolutional accelerator component, a large number of additional fetch operations must be performed to process convolutions with convolutional layers having 1×1 kernels.

[0057] Another option is to use fine-grained accelerator hardware. However, this approach involves considerable overhead in terms of control circuitry and software processing to use fine-grained hardware to process larger kernels. Fine-grained hardware is less efficient than coarse-grained accelerators in processing larger kernels. As mentioned above, such large kernels, such as 3×3 kernels, are also frequently used in CNNs.

[0058] The inventors have recognized that the hardware resources of coarse-grained convolution accelerators (e.g., convolution accelerators designed to efficiently handle kernels with a height and width of 3×3) can be reused or reconfigured to perform convolutions more efficiently by employing two different operating modes instead of shutting down the components to use kernels with smaller height and width dimensions (e.g., 1×1, 1×2, ... 1×N).

[0059] When processing convolutional layers with 1×1 kernels, only one row of feature data needs to be stored each time for convolution with the 1×1 kernel, so the feature data does not require a deep three-row feature row buffer. In addition, the size characteristics of the deeper feature row buffer 114 are more suitable for storing kernel data associated with the deep 1×1 kernel than the size characteristics of the shallower kernel buffer 116.

[0060] The inventors recognized that when processing convolutional layers with kernels having smaller height and width dimensions, the data types stored in the two buffers can be switched. A first operating mode can be used to process kernels with larger height and width dimensions (e.g., 3×3 kernels), and a second operating mode can be used to process kernels with smaller height and width dimensions (e.g., 1×1, 1×2, ..., 1×N). A conceptual diagram in Figure 13 illustrates two example operating modes.

[0061] In the first operating mode, labeled normal mode in Figure 13, feature data is stored in feature row buffer 114, and kernel data is stored in kernel buffer 116. The first operating mode can be used, for example, to process a typical 3×3 kernel in a normally operating manner. See above. Figure 10 The discussion.

[0062] In the second operating mode, labeled Deep Mode in Figure 13, feature data is stored in kernel buffer 116 instead of feature row buffer, and kernel data is stored in feature row buffer 114. For example, in the second operating mode, two sets of 24 different 1×1×128 kernels can be stored in feature row buffer 114, and three rows of feature data with a depth of 128 pixels can be stored in kernel data buffer 116. The second operating mode can be used, for example, to process 1×1 kernels, such as 1×1 kernels with a depth that provides increased efficiency when using the second operating mode instead of the first operating mode (e.g., a depth greater than the number of MAC units in a MAC cluster; in the example discussed, the kernel depth is 128, which is greater than 72).

[0063] To facilitate the first and second operating modes, Figure 9The system 100 shown includes a mode control circuitry 130. In a first operating mode, the mode control circuitry 130 controls the storage of feature data in a feature row buffer 114 and the transfer of feature data from the feature row buffer 114 to the MAC cluster 118, as well as the storage of kernel data in a kernel buffer 116 and the transfer of kernel data from the kernel buffer 116 to the MAC cluster 118. The operation of the mode control circuitry in the first operating mode can be performed to control the storage and transfer of feature and kernel data in a normally conventional manner. For example, as Figure 10 The concept diagram is shown. Although the mode control circuit device 130 is... Figure 9 While shown as a block in 13, the mode control circuitry can be implemented, for example, using control logic distributed through the convolution accelerator 112 and discrete components (e.g., multiplexers, existing buffers, etc.).

[0064] In the second operating mode, the mode control circuit device 130 controls the storage of kernel data in the feature row buffer 114 and the transfer of kernel data from the feature row buffer 114 to the MAC cluster 118, as well as the storage of feature data in the core buffer 116 and the transfer of feature data from the core buffer 116 to the MAC cluster 118. (References) Figure 14 - Figure 23 Examples of operation and configuration of an embodiment of the mode control circuit device 130 in the second operating mode are discussed.

[0065] Figure 9 Embodiments of system 100 may include more components than shown, may include fewer components than shown, may combine components, may separate components into sub-components, and various combinations thereof. For example, hardware accelerator 110 may include a DMA controller, mode control circuitry 130 may be divided into input mode control circuitry and output mode control circuitry, the input mode control circuitry controlling the storage of input feature data and kernel data into buffers 114, 116, the output mode control circuitry controlling the transfer of the stored input feature data from buffers 114, 116 to MAC cluster 118, all or part of mode control circuitry 130 may be shared by one or more convolutional accelerators 112, mode control circuitry 130 may be integrated into other control circuitry of the convolutional accelerator, and so on, and various combinations thereof.

[0066] Figure 14 This is a conceptual diagram illustrating an example of processing a convolutional layer with a 1×1 kernel by operating the convolutional accelerator in a second operating mode, where the kernel data is stored in a feature line buffer (e.g., Figure 9 The feature row buffer 114) and the pixels of the feature data are stored in the kernel buffer (e.g., Figure 9 In the kernel buffer 116).

[0067] The large single-port feature row buffer 114 is typically used to store 3 rows of feature data with a depth of 1024 elements, and it is configured to store 24 kernels with a depth of up to 128 kernel values. Therefore, 128 input channels are available, which is greater than the number mentioned in the reference above. Figure 9 The example configuration discussed has 72 output channels available. In the second operating mode, the control kernel buffer 116 is used to store three rows of feature data, and thus the three rows of feature data can be fed to the cluster of MAC units 118.

[0068] like Figure 14 As shown, kernels 1-4 are provided to a group of three clusters, each with four MAC units. One cluster of the three clusters with four MAC units computes the feature data and the output of line 1 of kernel 1-4, the second cluster computes the feature data and the output of line 2 of kernel 1-4, and the third cluster computes the feature data and the output of line 3 of kernel 1-4. Similarly, kernels 5-8 are provided to another group of three clusters, each with four MAC units, and each cluster computes the feature data and the output of the corresponding line of kernel 5-8. Kernels 9-12, 13-16, 17-20, and 21-24 are similarly provided to their respective clusters with four MAC units, and each cluster computes the corresponding line of feature data and the output of the corresponding kernel.

[0069] Compared to the normal operating mode, when processing convolutional layers with 1×1 kernels using the deep operating mode, the throughput is increased from 1 pixel depth per kernel per cycle to 3 pixels depth per kernel per cycle. Furthermore, when processing 1×1 kernels using the normal operating mode, using each kernel value in three MAC operations per cycle, rather than using each kernel value in one MAC operation per cycle, significantly improves the ratio of the number of extraction operations to the number of MAC operations performed (see [link to relevant documentation]). Figure 12 ).

[0070] Figure 15 This is a conceptual diagram illustrating an embodiment of the configuration of feature data path 220 and kernel data path 230 of a convolutional accelerator 112 operating in an operating mode such as a deep operating mode, wherein feature row data is stored in kernel buffer 116 and kernel data is stored in feature row buffer 114. An embodiment of feature data path 220 is described in... Figure 18 The embodiment of kernel data path 230 is shown in more detail below. Figure 19 It is shown in more detail below.

[0071] Figure 15 In the embodiment, the kernel buffer 116 includes six dual-port memories 216, as shown, six 16×96-bit dual-port buffers. In normal operating mode, the six dual-port memories 216 of the kernel buffer 116 store kernel data, which can be done in a conventional manner. In deep operating mode, four of the six dual-port memories 216 are included in the feature data path 220 and controlled by the feature load control circuitry 130f of the mode control circuitry device 130 to store feature row data.

[0072] As shown in the figure, four dual-port memories are organized into six 128×8-bit virtual buffers 116v in the feature row path 220, each virtual buffer 116v storing one row of feature data. Three rows of feature data can be fed to the MAC cluster 118 at a time, with each row provided to two of the six MAC unit clusters 118 by its corresponding virtual buffer. When feature data from three virtual buffers 116v is fed to the MAC cluster 118, the next set of calculated feature data can be loaded into the other three virtual buffers 116v. Therefore, three rows of feature data can be stored in the virtual buffers 116v of the kernel buffer memory 116 at a time in a ping-pong manner. The feature data loading control circuit 130f of the mode control circuitry 130 is included in the feature data path 220, and in operation, the feature data loading control circuit 130f controls the order in which pixels of feature data are stored in the virtual buffers 116v of the kernel buffer 116. As shown in the figure, the feature row path 220 includes a FIFO buffer and optional data processing circuitry.

[0073] In the second or deep operating mode, feature line buffer 114 is included in kernel data path 230 and controlled by kernel load control circuitry 130k of mode control circuitry 130 to store kernel data. The two remaining dual-port buffers 216 of kernel buffer 116 can advantageously be used in kernel data path 230 to store kernel data stored therein into and retrieve kernel data stored therein from feature line buffer 114. As shown, one of dual-port buffers 216a can be used to reduce the number of loops required to populate feature line buffer 114 with kernel data, and one of dual-port buffers 216b can be used to mirror the output of feature line buffer 114 to simulate the dual-port memory performance of single-port feature line buffer 114 (e.g., so that kernel data stored in dual-port buffer 216b can be used by MAC cluster 118 while subsequent kernel data is being loaded from dual-port buffer 216a into feature line buffer 114).

[0074] As shown in the figure, the kernel load control circuit 130k includes a filter circuit 130kfilt and a multiplexer circuit 130km. The filter circuit 130kfilt selects kernels from the kernel data path 230 during operation to load them into the feature line buffer 114 via the dual-port buffer 216a. The multiplexer circuit 130km controls the transmission of kernel data from the dual-port buffer 216b to the MAC cluster 118 during operation. Figure 15 The mode control circuit device 130 shown also includes a serialization circuit 130s, which serializes the output of the MAC cluster during operation to indicate the order in which MAC operations are performed.

[0075] Figure 16 This is a conceptual diagram illustrating features and kernel data flow paths in a deep operating mode of a convolutional accelerator according to an embodiment. Features flow in depth, and entire pixels are stored in each virtual buffer 116v of the kernel buffer 116. Three pixels are sent to the MAC clusters 118 at a time, with each pixel provided to a pair of six MAC clusters 118. For example, a multiplexer 130fm of the mode control circuitry 130 can be controlled to provide pixels 1-3 or pixels 4-6 to the corresponding MAC cluster pairs of the MAC clusters 118.

[0076] Kernels 1-24 are stored in feature line buffer 114, and all 24 kernels are sent to the corresponding MAC cluster in one cycle (see...). Figure 14 This generates three rows of the output feature map. Figure 17 This means that in Figure 16 The instance configuration is a conceptual diagram of the transmission of three pixels per cycle for a given kernel value. The same kernel strides three times in one cycle, obtaining three output values ​​for each kernel value, or obtaining 72 output values ​​for 24 kernels in one cycle.

[0077] Figure 18 This is a conceptual diagram illustrating a data path 220 of feature data in a convolutional accelerator operating in a deep operation mode according to an embodiment. The streaming engine 150 is capable of streaming, for example, 3 bytes or more of data in a transaction. As shown, the streaming engine streams pixel 1, depth 1 (…) of feature data in the first transaction T1. Figure 18 (1.1 in the middle), pixel 1, depth 2 ( Figure 18 (1.2) and pixel 1, depth 3 ( Figure 18 (1.3 in the text). Transaction T2 sends the next three depths of pixel 1, such as... Figure 18As shown in Figures 1.4, 1.5, and 1.6, transmission continues in a continuous stream until a set of pixels has been transmitted, as shown, until pixel W1 at depth 128 has been transmitted. Other pixel depths may be used. Pixels are stored in FIFO buffers and can be processed by data processing circuitry before being stored in the corresponding virtual buffers 116v by the feature load control circuitry 130f. As shown, each virtual buffer 116v provides three pixels at a time to the corresponding pair of the four MAC unit clusters. The three pixel values ​​(e.g., pixel 1, depth 1, pixel 2, depth 1, and pixel 3, depth 1) can be processed in a loop for each available kernel value (e.g., kernels 1-24), which provides a three-to-one improvement in throughput and improves the ratio of MAC computations per retrieved kernel value. As described above, while feature data from the three virtual buffers 116v is fed into the MAC cluster 118, the next set of computed feature data can be loaded into the other three virtual buffers 116v. Therefore, three rows of feature data can be stored in the virtual buffer 116v of the kernel buffer memory 116 in a ping-pong manner at one time.

[0078] Figure 19 This is a conceptual diagram illustrating the data path of kernel data 230 in a convolutional accelerator operating in a deep operation mode according to an embodiment. (Refer to above) Figure 15 The operations of the elements of the kernel data path are described. In each cycle, all 24 kernel values ​​stored in the dual-port memory 216b are provided to the corresponding MAC units in the 72 MAC units, with each kernel value being provided to 3 MAC units.

[0079] Figure 20 This is a conceptual diagram illustrating how, in a convolutional accelerator operating in a deep operation mode according to an embodiment, each kernel value (24 in the illustrated embodiment) is shared by three MAC units and each pixel of feature data is shared by a pair of MAC clusters to produce 72 output pixels per cycle. Figure 12 Compared to the previous implementation, the performance per cycle is three times better in processing convolutional layers with a 1×1 depth kernel, and each retrieved kernel value is used for three MAC operations instead of one. The additional control logic employed can increase the cost by a small amount (e.g., 2%) in terms of area.

[0080] Figure 14 - Figure 20 An example of a deep operation mode is shown in the context of convolution processing using a 1×1 kernel. Figure 21This is a conceptual diagram illustrating a convolution operation associated with a 1×2 kernel in a convolution accelerator operating in deep mode, according to an embodiment. The 1×2 kernel is decomposed into two 1×1 sub-kernels, and pixel values ​​are stepped. The first 1×1 sub-kernel is processed in a first deep mode cycle, using pixel values ​​1-3 to produce a first partial sum, and the second 1×1 sub-kernel is processed in a second deep mode cycle, using pixel values ​​2-4 to produce the accumulated output for the 1×2 kernel.

[0081] Figure 22 is a conceptual diagram illustrating a convolution operation associated with a 1×3 kernel in a convolution accelerator operating in deep mode, according to an embodiment. The 1×2 kernel is decomposed into three 1×1 sub-kernels, and pixel values ​​are stepped. The first 1×1 sub-kernel is processed in a first deep mode cycle, producing a first partial sum using pixel values ​​1–3; the second 1×1 sub-kernel is processed in a second deep mode cycle, producing a second partial sum using pixel values ​​2–4; and the third 1×1 sub-kernel is processed in a third deep mode cycle, producing the cumulative output of the 1×3 kernel using pixel values ​​3–5. This concept can be extended to convolution processing with 1×N kernels (e.g., to the available memory in the feature row buffer 114 for storing kernel depth). For example, in some embodiments, multiple convolution accelerators may be employed to process the sub-kernels.

[0082] Figure 23 This is a conceptual diagram illustrating a convolution operation associated with a 1×2 kernel in a convolution accelerator operating in a deep operation mode using an adder tree, according to an embodiment. As shown, each of the two sub-kernels is convolved with two pixel values ​​in one cycle, and the adder tree is used to combine the results, which reduces the waiting time associated with the processing results of the sub-kernels in the accumulation sequence cycle.

[0083] Figure 24 A logic flowchart is shown, generally illustrating an embodiment of a method 2400 for controlling a convolution accelerator in multiple operating modes, which can be used, for example, by convolution accelerator 112 as described above. Figure 9 - Figure 23 The mode control circuit device 130 discussed herein shall be used for execution. For convenience, reference will be made to... Figure 1 - Figure 23 Description method 2400.

[0084] Method 2400 begins at 2402 and proceeds to 2404. At 2404, method 2400 determines or selects the operating mode of the convolutional accelerator 112 when processing the convolutional layer. This can be done, for example, based on the size of the kernels to be processed by the convolutional accelerator 112 in the convolutional layer, the configuration of the convolutional accelerator 112 (e.g., characteristics of the feature row buffer 114, kernel buffer 116, etc.), and various combinations thereof. For example, if the convolutional accelerator 112 is optimized to process 3×3 kernels (e.g., based on the size and configuration of the cluster of kernel buffer 116 and MAC circuit 118), and the kernels to be processed are 2×2 or larger, a first operating mode, such as the normal operating mode, can be selected. On the other hand, if the convolutional accelerator 112 is optimized to process 3×3 kernels (e.g., based on the size and configuration of kernel buffer 116), and the kernels to be processed are 1×1 to 1×N kernels, where N is an integer greater than or equal to 1, a second operating mode, such as the deep operating mode, can be selected. Other factors, such as core depth, can also be considered. The operating mode can be determined by the main processor (e.g., Figure 9 The process 102) is performed and stored in the configuration register (e.g. Figure 10 In the configuration register 148). Method 2400 proceeds from 2404 to 2406.

[0085] At 2406, method 2400 determines, based on the determination at 2404, whether the selected operating mode is a first operating mode or a second operating mode, such as a normal operating mode or a deep operating mode. When it is determined at 2406 that the operating mode is the first operating mode, method 2400 proceeds from 2406 to 2408.

[0086] In step 2408, method 2400 stores the feature data in feature row buffer 114. This can be done in a conventional manner, such as as described in the reference above. Figure 10 As discussed in section 13. This method is implemented from 2408 to 2410.

[0087] At 2410, method 2400 stores the kernel data in kernel buffer 116. This can be done in a conventional manner, such as the reference above. Figure 10 As discussed in section 13. This method proceeds from 2410 to 2412.

[0088] At 2412, method 2400 transfers feature data from feature row buffer 114 to MAC cluster 118 and kernel data from kernel buffer 116 to MAC cluster 118. This can be done in a conventional manner, such as as described above. Figure 10 As discussed in section 13. This method proceeds from 2412 to 2414.

[0089] At 2414, method 2400 uses MAC cluster 118, which has the characteristic data and kernel data transmitted at 2412, to perform MAC operations. This can be done in a conventional manner, such as as described above. Figure 10 As discussed in section 13. This method proceeds from 2414 to 2416.

[0090] At 2416, method 2400 performs other processing operations associated with the convolutional layer, such as accumulating the results of batch data, serializing the output data, returning to 2408 to process subsequent batch data of the convolutional layer, transferring data to and from external memory, and various combinations thereof. This can be done in a conventional manner, such as as described above. Figure 8 The methods discussed in 10 and 13 are as follows. This method proceeds from 2416 to 2418.

[0091] At 2418, method 2400 performs other processing operations associated with the CNN, including convolutional layers, such as returning to 2404 to determine the operating mode of the next convolutional layer of the CNN, performing a merging operation of the next layer of the CNN, transferring data to and from external memory, and various combinations thereof.

[0092] When it is determined at 2406 that the operating mode is the second operating mode (e.g., deep kernel operating mode), method 2400 proceeds from 2406 to 2420.

[0093] In step 2420, method 2400 stores the feature data in kernel buffer 116. This can be achieved, for example, using reference to Figure 13 above. Figure 18 and Figure 20 Proceed as discussed. The method proceeds from 2420 to 2422.

[0094] At 2422, method 2400 stores the kernel data in the feature line buffer 114. This can be done, for example, as discussed above with reference to Figures 13-17, 19, and 20. The method proceeds from 2422 to 2424.

[0095] At 2424, method 2400 transfers feature data from kernel buffer 116 to MAC cluster 118, and transfers kernel data from feature row buffer 114 to MAC cluster 118. This can be done, for example, as discussed above with reference to Figures 13-20. The method proceeds from 2424 to 2426.

[0096] At 2426, method 2400 uses MAC cluster 118, which has the characteristic data and kernel data transmitted at 2424, to perform a MAC operation. This can be done in a normally conventional manner. The method proceeds from 2414 to 2416.

[0097] At 2428, method 2400 performs other processing operations associated with the convolutional layer, such as accumulating the results of batch data, serializing the output data (e.g., considering the order in which feature rows and kernel data were processed in 2426), and returning to 2420 to process subsequent batch data for the convolutional layer (see [link to previous section]). Figure 8 This includes transferring data to and from external memory, as well as various combinations thereof. The method proceeds from 2428 to 2418, where, as described above, other processing operations associated with the CNN can be performed.

[0098] Embodiments of the foregoing processes and methods may include Figure 24 Additional actions not shown may be omitted. Figure 24 All the actions shown can be performed in various orders. Figure 24 The actions shown can be combined, broken down into individual actions, and modified in various ways. For example, Figure 24 It can be modified to combine actions 2404 and 2406, execute actions 2408 and 2410 in parallel, and execute actions 2420 and 2422 in parallel, including processing actions that facilitate handling 1×N kernels in the second operating mode, where N is an integer greater than 1 (see example...). Figure 21 -23) and their various combinations.

[0099] In one embodiment, the convolution accelerator includes a feature row buffer, a kernel buffer separate from the feature row buffer, a multiply-accumulate (MAC) cluster, and mode control circuitry coupled to the feature row buffer, the kernel buffer, and the MAC cluster. In a first operating mode of the convolution accelerator, the mode control circuitry stores feature data in the feature row buffer, stores kernel data in the kernel buffer, transfers feature data from the feature row buffer to the MAC cluster, and transfers kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the convolution accelerator, the mode control circuitry again stores feature data in the kernel buffer, stores kernel data in the feature row buffer, transfers feature data from the kernel buffer to the MAC cluster, and transfers kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1.

[0100] In one embodiment, the mode control circuitry, in a first operating mode, stores three rows of feature row data with a depth of up to 1024 elements in a feature row buffer; and stores a 3×3 kernel in a kernel buffer. In one embodiment, the mode control circuitry, in a second operating mode, stores six rows of feature row data with a depth of up to 128 elements in a kernel buffer; and stores a 1×1 kernel in the feature row buffer. In one embodiment, the mode control circuitry, in the second operating mode, transfers three rows of feature row data from the kernel buffer to the MAC cluster in one cycle; and transfers 24 kernel data values ​​to the MAC cluster in the loop. In one embodiment, the MAC cluster generates 72 output values ​​in the cycle during operation.

[0101] In one embodiment, the feature row buffer is a single-port memory, and the kernel buffer includes multiple dual-port buffers. In one embodiment, the mode control circuitry, in a second operating mode,: stores feature row data in a first subset of the multiple dual-port buffers; and buffers kernel data in a second subset of the multiple dual-port buffers. In one embodiment, buffering kernel data in the second subset of the multiple dual-port buffers includes: storing kernel data in a first dual-port buffer of the second subset; transferring kernel data from the first dual-port buffer of the second subset to the feature row buffer; transferring kernel data from the feature row buffer to a second dual-port buffer of the second subset; and transferring kernel data from the second dual-port buffer of the second subset to the MAC cluster. In another embodiment, buffering kernel data in the second subset of the multiple dual-port buffers includes: transferring kernel data from the feature row buffer to a dual-port buffer of the dual-port buffer of the second subset; and transferring kernel data from the dual-port buffer of the second subset of the dual-port buffer to the MAC cluster. In one embodiment, in the second operating mode, the mode control circuitry serializes the output values ​​generated by the MAC cluster.

[0102] In one embodiment, the convolution accelerator includes a configuration register, and the mode control circuitry determines during operation whether to operate in a first operating mode or a second operating mode based on configuration parameters stored in the configuration register. In one embodiment, in the second operating mode, the kernel data has a size of 1×N, where N is an integer greater than or equal to 1.

[0103] In one embodiment, a system includes: a streaming engine that enables feature and kernel data streams in operation; and a convolution accelerator coupled to the streaming engine, wherein the convolution accelerator receives feature and kernel data streams from the streaming engine in operation. The convolution accelerator includes a feature row buffer, a kernel buffer, a multiply-accumulate cluster coupled to the feature row buffer and the kernel buffer, and a mode control circuitry coupled to the feature row buffer, the kernel buffer, and the MAC cluster. In a first operating mode of the convolution accelerator, the mode control circuitry stores feature data in the feature row buffer, stores kernel data in the kernel buffer, transfers feature data from the feature row buffer to the MAC cluster, and transfers kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the convolution accelerator, the mode control circuitry stores feature data in the kernel buffer, stores kernel data in the feature row buffer, transfers feature data from the kernel buffer to the MAC cluster, and transfers kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1.

[0104] In one embodiment, the mode control circuitry: in a first operating mode, stores three rows of feature row data with a depth of up to 1024 elements in a feature row buffer; and in a second operating mode, stores six rows of feature row data with a depth of up to 128 elements in a kernel buffer; and stores 1×N kernels in the feature row buffer, where N is an integer greater than or equal to 1. In one embodiment, the mode control circuitry, in the second operating mode, transfers three rows of feature row data from the kernel buffer to the MAC cluster in one cycle; and transfers 24 kernel data values ​​to the MAC cluster in a loop. In one embodiment, the feature row buffer is a single-port memory; the kernel buffer includes multiple dual-port buffers. In one embodiment, the mode control circuitry, in the second operating mode, stores feature row data in a first subset of the multiple dual-port buffers; and buffers kernel data in a second subset of the multiple dual-port buffers.

[0105] In one embodiment, a method includes streaming feature data and kernel data to a convolution accelerator, and convolving the streamed kernel data with the streamed feature data. The convolution includes, in a first operating mode of the convolution accelerator, storing feature data in a feature row buffer of the convolution accelerator, storing kernel data in a kernel buffer of the convolution accelerator, transferring feature data from the feature row buffer to the MAC cluster of the convolution accelerator, and transferring kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the convolution accelerator, the convolution includes storing feature data in a kernel buffer, storing kernel data in a feature row buffer, transferring feature data from the kernel buffer to the MAC cluster, and transferring kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1.

[0106] In one embodiment, a first operating mode includes storing three rows of feature row data with a depth of up to 1024 elements in a feature row buffer and storing 3×3 kernels in a kernel buffer; a second operating mode includes storing six rows of feature row data with a depth of up to 128 elements in a kernel buffer and storing 1×N kernels in the feature row buffer, where N is an integer greater than or equal to 1. In one embodiment, the kernel buffer includes multiple two-port buffers; and in the second operating mode: storing feature data in the kernel buffer includes storing feature data in a first subset of the multiple two-port buffers; storing kernel data in the feature row buffer includes buffering kernel data in a second subset of the multiple two-port buffers.

[0107] In one embodiment, the content of a non-transitory computer-readable medium is configured with a convolution accelerator having multiple operating modes to convolve streaming kernel data with streaming feature data. The convolution includes: in a first operating mode of the multiple operating modes of the convolution accelerator, storing feature data in a feature row buffer of the convolution accelerator, storing kernel data in a kernel buffer of the convolution accelerator, transferring feature data from the feature row buffer to the MAC cluster of the convolution accelerator, and transferring kernel data from the kernel buffer to the MAC cluster. In a second operating mode of the multiple operating modes of the convolution accelerator, the convolution includes storing feature data in a kernel buffer, storing kernel data in a feature row buffer, transferring feature data from the kernel buffer to the MAC cluster, and transferring kernel data from the feature row buffer to the MAC cluster. The second operating mode can be used to efficiently process 1×N kernels, where N is an integer greater than or equal to 1. In one embodiment, in a first operating mode, the convolution includes storing three rows of feature row data with a depth of up to 1024 elements in a feature row buffer and storing 3×3 kernels in a kernel buffer; and in a second operating mode, the convolution includes storing six rows of feature row data with a depth of up to 128 elements in a kernel buffer and storing 1×N kernels in the feature row buffer, where N is an integer greater than or equal to 1. In one embodiment, the kernel buffer includes multiple two-port buffers; and in the second operating mode: storing feature data in the kernel buffer includes storing feature data in a first subset of multiple two-port buffers; and storing kernel data in the feature row buffer includes kernel data buffered in a second subset of multiple two-port buffers. In one embodiment, the contents include instructions executed by the convolution accelerator.

[0108] Some embodiments may take the form of or include a computer program product. For example, according to one embodiment, a computer-readable medium is provided that includes a computer program adapted to perform one or more of the methods or functions described above. The medium may be a physical storage medium, such as a read-only memory (ROM) chip, or a disk, such as a digital multifunction disc (DVD-ROM), optical disc (CD-ROM), hard disk, memory, network, or a portable media article readable by a suitable drive or via a suitable connection, including one or more barcodes or other related codes encoded on one or more such computer-readable media and readable by a suitable reader device.

[0109] Furthermore, in some embodiments, some or all of these methods and / or functions may be implemented or provided in other ways, such as at least in part in firmware and / or hardware, including but not limited to one or more application-specific integrated circuits (ASICs), digital signal processors, discrete circuits, logic gates, standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and devices employing RFID technology and various combinations thereof.

[0110] The various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments based on the detailed description above. Generally, the terminology used in the following claims should not be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be interpreted to include all possible embodiments and the full scope of the authorized equivalents of these claims. Therefore, the claims are not limited to this disclosure.

Claims

1. A convolution accelerator, comprising: Feature line buffer; The kernel buffer is separate from the feature line buffer; Multiplicative MAC cluster; Configuration registers; as well as A mode control circuit device is coupled to the feature row buffer, the kernel buffer, the configuration register, and the MAC cluster, wherein the mode control circuit device: In the first operating mode of the convolution accelerator: The feature data is stored in the feature row buffer; Kernel data is stored in the kernel buffer; Transmit feature data from the feature row buffer to the MAC cluster; as well as Transferring kernel data from the kernel buffer to the MAC cluster; and In the second operating mode of the convolution accelerator: The feature data is stored in the kernel buffer; The kernel data is stored in the feature row buffer; Transmit feature data from the kernel buffer to the MAC cluster; as well as Kernel data is transferred from the feature row buffer to the MAC cluster. in: The feature row buffer is a single-port memory. The kernel buffer includes multiple dual-port buffers, and The mode control circuit device determines whether to operate in the first operating mode or the second operating mode based on the configuration parameters stored in the configuration register.

2. The convolution accelerator according to claim 1, wherein the mode control circuitry in the first operating mode: The feature row buffer stores three rows of feature row data with a depth of up to 1024 elements; and The kernel buffer stores 3×3 kernels.

3. The convolution accelerator according to claim 2, wherein the mode control circuitry is in the second operating mode: The kernel buffer stores six rows of feature row data with a depth of 128 elements; and A 1×1 kernel is stored in the feature row buffer.

4. The convolution accelerator according to claim 3, wherein the mode control circuitry is in the second operating mode: During the cycle, three lines of characteristic data are transferred from the kernel buffer to the MAC cluster; and During the cycle, 24 kernel data values ​​are transmitted to the MAC cluster.

5. The convolution accelerator of claim 4, wherein the MAC cluster generates 72 output values ​​during the cycle in operation.

6. The convolution accelerator of claim 1, wherein the mode control circuitry is in the second operating mode: The feature row data is stored in the first subset of the plurality of dual-port buffers; and Kernel data is buffered in the second subset of the plurality of dual-port buffers.

7. The convolution accelerator of claim 6, wherein buffering kernel data in the second subset of the plurality of dual-port buffers comprises: The kernel data is stored in the first dual-port buffer of the second subset; Transfer kernel data from the first dual-port buffer of the second subset to the feature row buffer; Transfer kernel data from the feature row buffer to the second dual-port buffer of the second subset; as well as Kernel data is transferred from the second dual-port buffer of the second subset to the MAC cluster.

8. The convolution accelerator of claim 6, wherein the buffered kernel data in the second subset of the plurality of dual-port buffers comprises: Transfer kernel data from the feature row buffer to the second subset of the dual-port buffer; as well as Kernel data is transferred from the second subset of the dual-port buffer to the MAC cluster.

9. The convolution accelerator of claim 1, wherein in the second operating mode, the mode control circuitry serializes the output values ​​generated by the MAC cluster.

10. The convolution accelerator of claim 1, wherein in the second operating mode, the kernel data has a size of 1×N, where N is an integer greater than or equal to 1.

11. An electronic system comprising: A streaming engine that streams features and kernel data during operation; as well as A convolutional accelerator, coupled to the streaming engine, wherein the convolutional accelerator receives, during operation, a stream of features and a stream of kernel data from the streaming engine, the convolutional accelerator comprising: Feature line buffer; The kernel buffer is separate from the feature row buffer; The multiply-accumulate MAC cluster is coupled to the feature row buffer and the kernel buffer; Configuration registers; and A mode control circuit device is coupled to the feature row buffer, the kernel buffer, the configuration register, and the MAC cluster, wherein the mode control circuit device: In the first operating mode of the convolution accelerator: The feature data is stored in the feature row buffer; Kernel data is stored in the kernel buffer; Transmitting feature data from the feature row buffer to the MAC cluster; and Transmitting kernel data from the kernel buffer to the MAC cluster; and In the second operating mode of the convolution accelerator: The feature data is stored in the kernel buffer; The kernel data is stored in the feature row buffer; Transmitting feature data from the kernel buffer to the MAC cluster; and Transmit kernel data from the feature row buffer to the MAC cluster in: The feature row buffer is a single-port memory. The kernel buffer includes multiple dual-port buffers, and The mode control circuit device determines whether to operate in the first operating mode or the second operating mode based on the configuration parameters stored in the configuration register.

12. The electronic system of claim 11, wherein the mode control circuit device: In the first operating mode: The feature row buffer stores three rows of feature row data with a depth of up to 1024 elements; and In the second operating mode: The kernel buffer stores six rows of feature row data with a depth of 128 elements; and The feature row buffer stores 1×N kernels, where N is an integer greater than or equal to 1.

13. The electronic system of claim 12, wherein the mode control circuitry is in the second operating mode: During the cycle, three lines of characteristic data are transferred from the kernel buffer to the MAC cluster; and During the cycle, 24 kernel data values ​​are transmitted to the MAC cluster.

14. The electronic system of claim 11, wherein the mode control circuitry is in the second operating mode: The feature row data is stored in the first subset of the plurality of dual-port buffers; and Kernel data is buffered in the second subset of the plurality of dual-port buffers.

15. A method for processing data, comprising: Streaming feature data and kernel data to the convolution accelerator according to any one of claims 1 to 10; as well as Convolving the streamed kernel data with the streamed feature data, the convolution including: In the first operating mode of the convolution accelerator: The feature data is stored in the feature row buffer of the convolution accelerator; The kernel data is stored in the kernel buffer of the convolution accelerator; Transferring feature data from the feature row buffer to the MAC cluster of the convolution accelerator; and Transferring kernel data from the kernel buffer to the MAC cluster; and In the second operating mode of the convolution accelerator: The feature data is stored in the kernel buffer; The kernel data is stored in the feature row buffer; Transferring feature data from the kernel buffer to the MAC cluster; and Kernel data is transferred from the feature row buffer to the MAC cluster.

16. The method of claim 15, wherein: The first operating mode includes storing three rows of feature row data with a depth of up to 1024 elements in the feature row buffer, and storing a 3×3 kernel in the kernel buffer; and The second operating mode includes storing six rows of feature row data with a depth of 128 elements in the kernel buffer, and storing 1×N kernels in the feature row buffer, where N is an integer greater than or equal to 1.

17. The method of claim 15, wherein: The kernel buffer includes multiple dual-port buffers; and In the second operating mode: Storing feature data in the kernel buffer includes storing feature data in a first subset of the plurality of dual-port buffers; as well as The kernel data is stored in the feature row buffer, which includes the buffered kernel data in a second subset of the plurality of dual-port buffers.

18. A non-transitory computer-readable medium having content configured with a convolution accelerator according to any one of claims 1 to 10, the convolution accelerator having a plurality of operating modes for convolving streaming kernel data with streaming feature data, the convolution comprising: In the first operating mode of the plurality of operating modes of the convolution accelerator: The feature data is stored in the feature row buffer of the convolution accelerator; The kernel data is stored in the kernel buffer of the convolution accelerator; Transferring feature data from the feature row buffer to the MAC cluster of the convolution accelerator; and Transferring kernel data from the kernel buffer to the MAC cluster; and In the second operating mode of the plurality of operating modes of the convolution accelerator: The feature data is stored in the kernel buffer; The kernel data is stored in the feature row buffer; Transferring feature data from the kernel buffer to the MAC cluster; and Kernel data is transferred from the feature row buffer to the MAC cluster.

19. The non-transitory computer-readable medium according to claim 18, wherein: In the first operating mode, the convolution includes storing three rows of feature row data with a depth of up to 1024 elements in the feature row buffer, and storing a 3×3 kernel in the kernel buffer; and In the second operating mode, the convolution includes storing six rows of feature row data with a depth of up to 128 elements in the kernel buffer, and storing a 1×N kernel in the feature row buffer, where N is an integer greater than or equal to 1.

20. The non-transitory computer-readable medium according to claim 18, wherein: The kernel buffer includes multiple dual-port buffers; and In the second operating mode: Storing feature data in the kernel buffer includes storing feature data in a first subset of the plurality of dual-port buffers; as well as Storing kernel data in the feature row buffer includes buffering kernel data in a second subset of the plurality of dual-port buffers.

21. The non-transitory computer-readable medium of claim 18, wherein the content includes instructions executed by the convolution accelerator.

Images