Method and apparatus for performing deep learning operations

Abstract

Methods and apparatuses for performing deep learning operations are provided. The computing apparatus includes an adder-tree-based tensor kernel and a multiplier-accumulator (MAC)-based vector kernel. The adder-tree-based tensor kernel is configured to perform tensor operations, and the multiplier-accumulator (MAC)-based vector kernel is configured to perform vector operations using the output of the tensor kernel as input.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0167970, filed on December 4, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to a method and apparatus for performing deep learning operations, and more specifically, to a computing apparatus configured with a multiplier- and accumulator-based (MAC) coprocessor in an adder tree-based artificial neural network accelerator. Background Technology

[0003] Artificial neural networks (ANNs) are implemented based on computational architectures. Due to advancements in ANN technology, active research is underway to utilize ANNs in various types of electronic systems to analyze input data and extract useful information.

[0004] Devices used to process ANNs require massive computations on complex input data. Most artificial intelligence (AI) applications provide complex but slow-responding services via network connections between servers and clients. This is because ANNs (the key algorithms in deep learning) typically require 1000 times or more of computational operations than a general-purpose application processor (AP).

[0005] Therefore, an architecture that can efficiently meet such high computational requirements is needed. Summary of the Invention

[0006] The present invention is provided to introduce, in a simplified form, the selection of concepts further described in the following detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0007] In one general aspect, a device for performing deep learning operations includes: a tensor kernel configured to perform tensor operations, the tensor kernel being based on an adder tree; and a vector kernel configured to perform vector operations using the output of the tensor kernel as input, the vector kernel being based on a multiplier and accumulator (MAC).

[0008] Tensor operations can include matrix-matrix multiplication, and vector operations can include any one or any combination of vector-matrix multiplication, vector-vector multiplication, and element-wise operations.

[0009] The vector kernel may include a MAC-based arithmetic logic unit (ALU) configured to perform vector operations.

[0010] A vector kernel may include: a functional unit configured to perform one or both of pooling operations and nonlinear function operations.

[0011] Functional units may include lookup tables for performing one or both of pooling operations and nonlinear function operations.

[0012] The vector kernel may include a weight buffer, which is configured to store weight data.

[0013] The device may also include a local buffer configured to store data so that the vector kernel can reuse the output of the tensor kernel to perform vector operations.

[0014] The vector kernel can be configured to perform the first vector operation while the tensor kernel performs the second tensor operation, using the output of the first tensor operation as input.

[0015] The vector kernel may include: a functional unit configured to perform one or both of pooling operations and nonlinear function operations; a MAC-based ALU configured to perform vector operations; a weight buffer configured to store weight data; and a first multiplexer configured to select at least one of the output of the tensor kernel and the output of the ALU as the input of the functional unit.

[0016] The vector kernel may also include a second multiplexer configured to select at least one of the output of the functional unit and the output of the first multiplexer as the input of the ALU.

[0017] The vector kernel may also include a third multiplexer configured to select at least one of the outputs of the functional unit and the ALU as the output of the vector kernel.

[0018] Tensor kernels can be configured to perform traversals of tensor operations in units of blocks used to perform vector operations.

[0019] Tensor kernels can be configured to perform convolution operations. Vector kernels may include: functional units configured to perform one or both of pooling operations and nonlinear function operations; a MAC-based ALU configured to perform vector operations; and a weight buffer configured to store weight data. The functional units can be configured to receive the output of the convolution operation as input and perform a first activation function operation; the ALU can be configured to perform a depthwise convolution operation between the weight data and the result of the first activation function operation; and the functional units can be configured to receive the result of the depthwise convolution operation as input and perform a second activation function operation.

[0020] In another general aspect, a method for performing deep learning operations includes: performing tensor operations using a tensor kernel based on adder trees; and performing vector operations using the output of the tensor kernel as input via a MAC-based vector kernel.

[0021] Tensor operations can include matrix-matrix multiplication, and vector operations can include any one or any combination of vector-matrix multiplication, vector-vector multiplication, and element-wise operations.

[0022] The steps of performing tensor operations may include performing convolution operations. The steps of performing vector operations may include: receiving the output of the convolution operation as input and performing a first activation function operation; performing a depthwise convolution operation between the weight data and the result of the first activation function operation; and receiving the result of the depthwise convolution operation as input and performing a second activation function operation.

[0023] The steps for performing tensor operations may include: performing a traversal of tensor operations in units of blocks used for vector operations.

[0024] In another general aspect, a device includes: one or more processors configured to: perform a first tensor operation and a second tensor operation after the first tensor operation, and perform vector operations using the output of the first tensor operation as input while simultaneously performing the second tensor operation.

[0025] The one or more processors may include: an adder tree-based artificial neural network (ANN) accelerator configured to perform a first tensor operation and a second tensor operation; and a multiplier and accumulator (MAC)-based processor configured to perform vector operations.

[0026] The first tensor operation can be a first convolution operation, the second tensor operation can be a second convolution operation, and the vector operation can be a depthwise convolution operation.

[0027] The one or more processors can be configured to perform an activation function operation on the result of the first convolution operation to generate the output of the first tensor operation.

[0028] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0029] Figure 1A An example of a method for performing deep learning operations using an artificial neural network (ANN) is shown.

[0030] Figure 1B An example of data and filters provided as input feature maps in deep learning operations is shown.

[0031] Figure 1C This paper demonstrates a method for performing deep learning operations using multiple multipliers and accumulators (MACs) according to existing technologies.

[0032] Figure 1D This paper illustrates a method for performing deep learning operations using adder trees, based on existing techniques.

[0033] Figure 2 An example of a hardware implementation of a computing device is shown.

[0034] Figure 3 An example of a hardware implementation of a vector core is shown.

[0035] Figure 4 An example of how a computing device is operated is shown.

[0036] Figure 5A and Figure 5B An example of using computing devices to process depth-wise convolution blocks is shown.

[0037] Figure 6A and Figure 6B An example of a method to reduce the idle time of vector operations is shown.

[0038] Figure 7A and Figure 7B An example of a method for performing vector operations is shown.

[0039] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0040] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein; rather, the order of operations may be altered, as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known in the art upon understanding this disclosure may be omitted.

[0041] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will be clear upon understanding the disclosure of this application.

[0042] The following structural or functional descriptions of the examples disclosed in this disclosure are intended for illustrative purposes only, and the examples may be implemented in various forms. These examples are not intended to be limiting, but various modifications, equivalents, and alternatives are also intended to be covered within the scope of the claims.

[0043] Although the terms "first" or "second" are used to describe various components, the components are not limited by these terms. These terms should only be used to distinguish one component from another. For example, within the scope of the claims based on the concept of this disclosure, a "first" component may be referred to as a "second" component, or similarly, a "second" component may be referred to as a "first" component.

[0044] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with the other component, or there may be an intermediate component.

[0045] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As used herein, the terms "comprising," "including," and "having" indicate the presence of the stated features, quantities, operations, elements, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, elements, components, and / or combinations thereof. The use of the term "may" (e.g., regarding what an example or embodiment may include or implement) with respect to examples or embodiments indicates the presence of at least one example or embodiment that includes or implements such features, while all examples are not limited thereto.

[0046] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as those terms commonly understood upon understanding this disclosure. Unless otherwise defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning matching the meaning in the relevant field and in the context of this disclosure, and shall not be interpreted as having an idealized or overly formal meaning.

[0047] When describing the examples with reference to the accompanying drawings, the same reference numerals denote the same constituent elements, and repetitive descriptions associated with them will be omitted. In the description of the examples, descriptions of well-known related structures or functions will be omitted where such detailed descriptions would lead to a vague interpretation of this disclosure.

[0048] Figure 1A An example of a method for performing deep learning operations using an artificial neural network (ANN) is shown.

[0049] Artificial intelligence (AI) algorithms, including deep learning, can feed input data 10 into an ANN and learn output data 30 through operations (e.g., convolution). An ANN can be a computational architecture obtained by simulating a biological brain. In an ANN, nodes corresponding to neurons in the brain can be connected to each other and can operate together to process input data. Various types of neural networks can include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks (DBNs), or restricted Boltzmann machines (RBMs). In a feedforward neural network, neurons can have links to other neurons. These links can be extended in a single direction (e.g., the forward direction) through the neural network.

[0050] Figure 1A The diagram illustrates a structure where input data 10 is fed into an ANN and output data 30 is output through the ANN. The ANN may include at least one layer and may be, for example, a CNN 20. The ANN may be, for example, a deep neural network (DNN) comprising at least two layers.

[0051] CNN 20 can be used to extract "features" (e.g., boundaries, lines, or colors) from input data 10. CNN 20 can include multiple layers. Each layer can receive data, process the data input to the corresponding layer, and generate data to be output from the corresponding layer. The data output from the layer can be a feature map generated by performing a convolution operation between the image or feature map input to CNN 20 and the weight values ​​of at least one filter. The initial layers of CNN 20 can operate to extract relatively low-level features (e.g., edges or gradients) from the input (such as image data). Subsequent layers of CNN 20 can progressively extract more complex features (e.g., eyes or nose in an image).

[0052] Figure 1B An example of data and filters provided as input feature maps in deep learning operations is shown.

[0053] Reference Figure 1B The input feature map 100 can be a set of numerical data or pixel values ​​of an image input to the ANN, but is not limited to these. Figure 1BIn this example, the input feature map 100 can be defined by the pixel values ​​of the target image to be trained using the ANN. For example, the input feature map 100 can have “256×256” pixels and a depth of “K”. However, the above values ​​are just examples, and the size of the pixels in the input feature map 100 is not limited to these.

[0054] N filters (e.g., filters 110-1 to 110-n) can be formed. Each of filters 110-1 to 110-n may include n×n weight values. For example, each of filters 110-1 to 110-n may have 3×3 pixels and K depth values. However, the above-described size of each of filters 110-1 to 110-n is merely an example and is not limited thereto.

[0055] Figure 1C This demonstrates a method for performing deep learning operations using multiple multipliers and accumulators (MACs) according to existing techniques, and Figure 1D This illustrates a method for performing deep learning operations using adder trees, based on existing techniques. Figure 1C and Figure 1D In this context, REG can represent a register.

[0056] Reference Figure 1D A computational device based on adder trees may include multiple multipliers that compute the product of two sets of data (e.g., weights and an input feature map IFM), adders that compute the sum of the outputs of two neighboring multipliers or the sum of the outputs of two neighboring adders, and a single accumulator that accumulates and sums the final output data. Because a small number of accumulators (e.g., a single accumulator) are used, a computational device based on adder trees can perform convolution operations with low power.

[0057] Reference Figure 1C A MAC-based computing device may include multiple processing elements (PEs) and can perform a relatively large number of operations simultaneously by inputting input data into each of the PEs. Each PE in a MAC-based computing device may be configured with a MAC that performs the operation ACC = ACC + (A × B).

[0058] In deep learning operations, operations with different characteristics (e.g., computationally constrained operations that account for a high proportion of computation time in the total execution time of the deep learning operation, and memory-constrained operations that account for a high proportion of memory access time in the total execution time) are mixed. For example, tensor operations (e.g., matrix-matrix multiplication) may have computationally constrained characteristics that account for a high proportion of computation time in the total execution time, and vector operations (e.g., vector-matrix multiplication, vector-vector multiplication, or element-wise operations) may have memory-constrained characteristics that account for a high proportion of memory access time in the total execution time.

[0059] To support high-precision and more complex and diverse applications, the requirements for vector operations in ANNs (such as interpolation, estimation of various types of nonlinear functions, or depthwise convolution) are increasing.

[0060] However, existing computing devices with adder tree structures are inefficient due to low utilization when memory-constrained operations are performed. For example, depthwise convolution operations can be performed on each input feature map for each input channel. However, if a computing device with an adder tree structure is used to perform depthwise convolution operations, resource utilization is reduced because of unused multipliers. Furthermore, since memory microfabrication speed is lower than logic microfabrication speed, logic density can be relatively increased. Therefore, the insufficient memory bandwidth in adder tree-based computing devices can be expected to worsen in the future.

[0061] MAC-based computing devices may include accumulators for each MAC unit, thus making them suitable for performing vector operations. For example, a MAC-based computing device might be suitable for performing depthwise convolution operations per channel. However, because MAC-based computing devices require accumulators for all MAC units, a relatively large amount of power is consumed. For example, when an adder-tree-based computing device performs a regular convolution operation, only a single accumulator is needed. However, when a MAC-based computing device performs a regular convolution operation, all accumulators operate, leading to reduced power efficiency. Therefore, MAC-based computing devices may be suitable for vector operations (e.g., parallel data processing operations such as depthwise convolution operations) but may not be suitable for tensor operations (e.g., regular convolution operations).

[0062] In one example based on this disclosure, a device for performing deep learning operations (hereinafter referred to as a "computing device") may include both an adder-tree-based ANN accelerator and a MAC-based coprocessor. The adder-tree-based ANN accelerator can be used to perform tensor operations, and the MAC-based coprocessor can be used to efficiently perform vector operations. In the following examples, examples of the operation methods of the computing device will be provided. Figures 2 to 7B It was described in detail.

[0063] Figure 2 An example of a hardware implementation of a computing device based on this disclosure is shown.

[0064] Reference Figure 2 The computing device may include a tensor kernel 210 and a vector kernel 220. The computing device may also include a local buffer 230, an on-chip memory 240, and an instruction fetch and control unit 250 (hereinafter referred to as "controller 250"). Although in Figure 2 The individual components of the computing device are shown separately to indicate that the components are functionally and logically separable, but this does not necessarily mean that the components are implemented as physically separate components or code.

[0065] Tensor core 210 can be an adder tree-based ANN accelerator that performs tensor operations, and vector core 220 can be a MAC-based coprocessor that performs vector operations.

[0066] For vector operations that use the output of tensor operations as input, the computing device can perform vector operations using the output of the tensor operations as input without writing-back the output of the tensor operations to the on-chip memory 240, instead of performing the vector operations after writing back the output of the tensor operations to the on-chip memory 240. Therefore, the computing device can reduce the memory bandwidth requirements of vector operations and improve the utilization of computing resources.

[0067] The computing device may include a local buffer 230 for reusing data. Data reuse can indicate that operations are performed by reusing pre-loaded data (e.g., weights or input feature maps). By reusing data, the number of times data is loaded and the number of operations performed can be reduced. For example, the local buffer 230 may be configured to store data such that vector kernel 220 can reuse the output of tensor kernel 210 to perform vector operations.

[0068] Figure 3 An example of a hardware implementation based on the vector kernel of this disclosure is shown.

[0069] Reference Figure 3The vector kernel 220 may include a functional unit 310, a weight buffer 320, and an arithmetic logic unit (ALU) 330. Although in Figure 3 The individual components of vector kernel 220 are shown separately to indicate that the components are functionally and logically separable, but this does not necessarily mean that the components are implemented as physically separate components or code.

[0070] Functional unit 310 can perform one or both of pooling operations and nonlinear function operations. Functional unit 310 may include a lookup table to perform one or both of pooling operations and nonlinear function operations.

[0071] The weight buffer 320 can store weight data.

[0072] The ALU 330 can be a MAC-based arithmetic logic device that performs vector operations.

[0073] Furthermore, the vector kernel 220 may also include a first multiplexer 340, a second multiplexer 350, and a third multiplexer 360. The first multiplexer 340 is configured to select at least one of the output of the tensor kernel 210 and the output of the ALU 330 as the input of the functional unit 310. The second multiplexer 350 is configured to select at least one of the output of the functional unit 310 and the output of the first multiplexer 340 as the input of the ALU 330. The third multiplexer 360 is configured to select at least one of the output of the functional unit 310 and the output of the ALU 330 as the output of the vector kernel 220. Referring below... Figures 4 to 7B Further examples of how to operate a computing device including tensor kernel 210 and vector kernel 220 are described.

[0074] Figure 4 An example of an operation method for a computing device based on this disclosure is shown.

[0075] Reference Figure 4 The left box 410 shows the computation time and memory access time required when only the tensor kernel is used to perform tensor and vector operations, and the right box 420 shows the computation time and memory access time required when a computing device including tensor kernels and vector kernels, according to the example, is used to perform tensor and vector operations.

[0076] exist Figure 4 In this context, the operations are assumed to include, for example, first tensor operations ①, second tensor operations ②, third tensor operations ③, and first vector operations. Second vector operation The output of the first tensor operation is the input of the first vector operation, and the output of the second tensor operation is the input of the second vector operation.

[0077] In a computing device, vector kernel 220 and tensor kernel 210 can perform operations in parallel, thus reducing the total computation time by hiding the computation time of vector operations.

[0078] In one example, referring to left box 410, a computing device using only tensor kernels can only perform vector operations after tensor operations have finished, and can only perform the next tensor operation after the vector operation has finished. For example, a computing device using tensor kernels may inevitably only perform the second tensor operation after the first vector operation has finished.

[0079] In another example, referring to right box 420, since the computing device independently includes a tensor kernel 210 configured to perform tensor operations and a vector kernel 220 configured to perform vector operations, the vector kernel 220, together with the tensor kernel 210, can perform operations in parallel (e.g., simultaneously). For example, the tensor kernel 210 can perform a second tensor operation while the vector kernel 220 is performing a first vector operation. Furthermore, the tensor kernel 210 can perform a third tensor operation while the vector kernel 220 is performing a second vector operation.

[0080] The computing device can perform vector operations using a vector kernel 220 optimized for vector operations, thus improving hardware utilization compared to using only a tensor kernel to perform operations.

[0081] Furthermore, the computing device can use the output of tensor operations as input to vector operations to perform vector operations, instead of performing vector operations after writing the output of tensor operations back to on-chip memory 240. As a result, the computing device can reduce on-chip memory access power consumption. Additionally, the computing device can reduce memory access idle time for tensor operations, which can lead to a reduction in the total time used to perform operations.

[0082] Referring to box 410, it is inevitable that a computing device using only a tensor kernel will write back the output of the tensor operation to on-chip memory 240 and then perform vector operations. For example, a computing device using only a tensor kernel will write back the output of the first tensor operation to on-chip memory 240 and then perform the first vector operation.

[0083] Referring to right box 420, the computing device can use the output of a tensor operation as the input of a vector operation to perform a vector operation, instead of writing the output of the tensor operation back to on-chip memory 240. For example, the computing device can use the output of a first tensor operation as the input of a first vector operation, and can use the output of a second tensor operation as the input of a second vector operation.

[0084] Figure 5A and Figure 5BAn example of processing deep convolutional blocks using a computing device based on this disclosure is shown.

[0085] Reference Figure 5A The depthwise convolution block may include convolution operation 510 (e.g., 1×1 convolution operation), depthwise convolution operation 520 (e.g., 3×3 depthwise convolution operation), and convolution operation 530 (e.g., 1×1 convolution operation). Convolution operations 510 and 530 may be computationally restricted tensor operations, and depthwise convolution operation 520 may be memory-restricted vector operations.

[0086] For example, the output obtained by performing an activation function operation (e.g., a rectified linear unit (ReLU) function operation) on the result of convolution operation 510 can be the input of depthwise convolution operation 520. The output obtained by performing an activation function operation (e.g., a ReLU function operation) on the result of depthwise convolution operation 520 can be the input of convolution operation 530.

[0087] The tensor kernel 210 of the computing device can perform convolution operation 510, and the vector kernel 220 can receive the output of convolution operation 510 as input. When the output of convolution operation 510 is received as input to the vector kernel 220, the functional unit 310 of the vector kernel 220 can perform a first activation function operation on the output of convolution operation 510.

[0088] The ALU 330 of the vector kernel 220 can receive weight data stored in the weight buffer 320, receive the result of the first activation function operation from the functional unit 310, and perform a depthwise convolution operation 520 between the weight data and the result of the first activation function operation.

[0089] Functional unit 310 can receive the result of depthwise convolution operation 520 as input, and can perform a second activation function operation. Vector kernel 220 can write back the result of the second activation function operation to on-chip memory 240.

[0090] Tensor kernel 210 can read the result of the second activation function operation and can perform convolution operation 530.

[0091] Reference Figure 5B The left box 540 shows the computation time and memory access time required when only tensor kernels are used to process deep convolution blocks, and the right box 550 shows the computation time and memory access time required when a computing device including tensor kernels and vector kernels, according to the example, is used to process deep convolution blocks.

[0092] The computing device can perform vector operations using a vector kernel 220 optimized for vector operations, thus improving hardware utilization compared to using only a tensor kernel. For example, the computing device can perform pooling operations and nonlinear function operations using the functional unit 310 of the vector kernel 220, and can perform depthwise convolution operations using the ALU 330 of the vector kernel 220.

[0093] Furthermore, the computing device can use the output of tensor operations as input to vector operations to perform vector operations, instead of performing vector operations after writing the output of tensor operations back to on-chip memory 240. As a result, the computing device can reduce on-chip memory access power consumption. Additionally, the computing device can reduce memory access idle time for tensor operations, which can lead to a reduction in the total time used to perform operations.

[0094] Referring to box 540, it is inevitable that a computing device using only tensor kernels will write back the output of tensor operations to on-chip memory 240 before performing vector operations. For example, a computing device using only tensor kernels will write back the output of convolution operation 510 to on-chip memory 240 before performing depthwise convolution operation 520.

[0095] Referring to right box 550, the computing device can use the output of tensor operations as input to vector operations to perform vector operations without first writing the output of the tensor operations back to on-chip memory 240. For example, the computing device can use the output of convolution operation 510 as input to depthwise convolution operation 520.

[0096] Figure 6A and Figure 6B An example of a method for reducing idle time in vector operations based on this disclosure is shown.

[0097] Reference Figure 6A 1×1 convolution operations and 3×3 depthwise convolution operations can be performed consecutively to accelerate deep convolution blocks within the vector kernel. For example, when a 1×1 convolution operation is performed in the row or column direction, idle time may occur until the 3×3 input feature map is ready to perform the 3×3 depthwise convolution operation. In this example, access time to on-chip memory and / or local buffers may increase for 3×3 depthwise convolution operations due to reduced data reusability of local buffers. In the following description, for ease of explanation, the description is based on 1×1 convolution operations and 3×3 depthwise convolution operations; however, the examples can also be applied to blocks of other sizes.

[0098] Reference Figure 6BTensor kernel 210 can perform traversal of tensor operations in blocks used for vector operations. For example, when performing a depthwise convolution operation in 3×3 blocks, tensor kernel 210 can traverse in 3×3 blocks while performing a 1×1 convolution operation to minimize idle time before preparing the input for the 3×3 depthwise convolution operation and maximize the reusability of the input feature map for the 3×3 depthwise convolution operation in local buffer 230. Therefore, the total time and power consumption for performing operations can be reduced by minimizing the access time of local buffer 230 and on-chip memory 240.

[0099] Figure 7A and Figure 7B An example of a method for performing vector operations based on this disclosure is shown.

[0100] Reference Figure 7A The vector kernel 220 can perform vector operations, such as max pooling and / or min pooling and / or nonlinear function operations.

[0101] Vector kernel 220 can use functional unit 310 to perform maximum comparison and / or minimum comparison operations for max pooling and / or min pooling operations. For example, vector kernel 220 can receive input data from an external source, perform maximum comparison and / or minimum comparison operations based on the current input and the maximum and / or minimum values ​​detected in functional unit 310 in previous cycles, and store the new maximum and / or minimum values ​​in the second multiplexer 350. When maximum comparison and / or minimum comparison operations have been performed on all elements of the pooling window, vector kernel 220 can output the final max pooling result and / or min pooling result.

[0102] Furthermore, functional unit 310 may include a lookup table to efficiently provide basic nonlinear function (e.g., ReLU, ReLU6, or PReLU) operations. For example, vector kernel 220 may receive input data from an external source, search the stored lookup table from functional unit 310, and perform nonlinear function operations.

[0103] Reference Figure 7B The vector kernel 220 can use interpolation and / or linear approximation operations to perform average pooling operations and nonlinear function operations.

[0104] Vector kernel 220 can use ALU 330 to perform average pooling operations. For example, vector kernel 220 can receive input data from an external source and can perform accumulation operations in ALU 330 after bypassing functional unit 310. When accumulation operations have been performed on all elements of the pooling window, vector kernel 220 can output the final average pooling result obtained by dividing the accumulation result by the size of the pooling window.

[0105] Furthermore, vector core 220 can perform nonlinear function operations using interpolation and / or linear approximation operations via ALU 330. For example, vector core 220 can receive input data from an external source and can perform the operation ACC = ACC + (A × B) in ALU 330 after bypassing functional unit 310. When the operation ACC = ACC + (A × B) has been performed on all blocks, vector core 220 can output the final result.

[0106] The devices, units, modules, apparatuses, and other components described herein are implemented using hardware components. For example, tensor kernel 210, vector kernel 220, local buffer 230, on-chip memory 240, instruction fetch and control unit 250, functional unit 310, weight buffer 320, arithmetic logic unit (ALU) 330, and multiplexers 340, 350, and 360 can all be implemented using hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more of the hardware components performing the operations described herein are implemented using computing hardware (e.g., one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For the sake of brevity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors or processors and controllers, and one or more other hardware components may be implemented by one or more other processors or additional processors and additional controllers. One or more processors or processors and controllers may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0107] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors or a processor and a controller, and one or more other operations may be executed by one or more other processors or additional processors and additional controllers. One or more processors or a processor and a controller may execute a single operation or two or more operations.

[0108] Instructions or software for controlling one or more processors or computers to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write the instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0109] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), flash memory, card-type storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files, and data structures to one or more processors or computers, enabling one or more processors or computers to execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0110] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if the components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents will be construed as included in the disclosure.

Claims

1. An apparatus for performing deep learning operations, the apparatus comprising: Tensor kernels based on adder trees are configured to perform tensor operations; as well as Vector kernels based on multipliers and accumulators are configured to perform vector operations using the output of the tensor kernel as input. Tensor operations include matrix-matrix multiplication, and vector operations include any one or any combination of vector-matrix multiplication, vector-vector multiplication, and element-wise operations. The vector kernel is configured to perform the first vector operation using the output of the first tensor operation as input while the tensor kernel performs the second tensor operation, wherein the first tensor operation is performed before the second tensor operation. The vector kernel includes: The functional unit is configured to perform one or both of pooling operations and nonlinear function operations; Arithmetic logic units based on multipliers and accumulators are configured to perform vector operations; A weight buffer is configured to store weight data; and The first multiplexer is configured to select at least one of the output of the tensor kernel and the output of the arithmetic logic unit as the input of the functional unit.

2. The device according to claim 1, wherein, The functional unit includes a lookup table for performing one or both of pooling operations and nonlinear function operations.

3. The device according to claim 1, further comprising: A local buffer is configured to store data so that the vector kernel can reuse the output of the tensor kernel to perform vector operations.

4. The device according to claim 1, wherein, The vector kernel also includes a second multiplexer configured to select at least one of the output of the functional unit and the output of the first multiplexer as the input of the arithmetic logic unit.

5. The device according to claim 4, wherein, The vector kernel also includes a third multiplexer configured to select at least one of the outputs of the functional unit and the arithmetic logic unit as the output of the vector kernel.

6. The device according to any one of claims 1 to 5, wherein, The tensor kernel is configured to perform traversal of tensor operations in units of blocks used to perform vector operations.

7. The device according to claim 1, wherein, The tensor kernel is configured to perform convolution operations. The functional unit is configured to: receive the output of the convolution operation as input and perform a first activation function operation. The arithmetic logic unit is configured to perform a depthwise convolution operation between the weight data and the result of the first activation function operation, and The functional unit is configured to receive the result of the depthwise convolution operation as input and perform a second activation function operation.

8. A method for performing deep learning operations, the method comprising: Tensor operations are performed using tensor kernels based on adder trees; Output the tensor kernel to a vector kernel based on multipliers and accumulators; as well as Vector operations are performed using the output of a tensor kernel as input through a vector kernel. Tensor operations include matrix-matrix multiplication, and vector operations include any one or any combination of vector-matrix multiplication, vector-vector multiplication, and element-wise operations. The vector kernel is configured to perform the first vector operation using the output of the first tensor operation as input while the tensor kernel performs the second tensor operation, wherein the first tensor operation is performed before the second tensor operation. The vector kernel includes: The functional unit is configured to perform one or both of pooling operations and nonlinear function operations; Arithmetic logic units based on multipliers and accumulators are configured to perform vector operations; A weight buffer is configured to store weight data; and The first multiplexer is configured to select at least one of the output of the tensor kernel and the output of the arithmetic logic unit as the input of the functional unit.

9. The method according to claim 8, wherein, The steps for performing tensor operations include performing convolution operations, and The steps for performing vector operations include: It receives the output of the convolution operation as input and performs the first activation function operation; Perform a depthwise convolution operation between the weighted data and the result of the first activation function; and It receives the result of the depthwise convolution operation as input and performs the second activation function operation.

10. The method according to any one of claims 8 to 9, wherein, The steps for performing tensor operations include: performing a traversal of tensor operations in units of blocks used for vector operations.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 8 to 10.

12. A computing device, the computing device comprising: The tensor kernel based on the adder tree is configured to perform the first tensor operation and then perform the second tensor operation after the first tensor operation. as well as The vector kernel based on multipliers and accumulators is configured to perform vector operations using the output of the first tensor operation as input while the tensor kernel performs the second tensor operation.

13. The computing device according to claim 12, wherein, The tensor kernel includes: an adder-tree-based artificial neural network accelerator configured to perform first tensor operations and second tensor operations; and The vector kernel comprises a processor based on multipliers and accumulators, configured to perform vector operations.

14. The computing device according to claim 12, wherein, The first tensor operation is the first convolution operation, the second tensor operation is the second convolution operation, and the vector operation is a depthwise convolution operation.

15. The computing device according to claim 14, wherein, The tensor kernel is configured to perform an activation function operation on the result of the first convolution operation to generate the output of the first tensor operation.

Images