Method and electronic device for processing neural network

Abstract

Disclosed are a method and electronic device for processing a neural network. The electronic device includes: a host processor configured to receive a request for an accelerator to process the neural network, generate an instruction executable by the accelerator in response to the request, and send the instruction to the accelerator; and an accelerator configured to execute the neural network in response to the instruction and send a result of executing the neural network to the host processor, wherein the accelerator is configured to detect whether an operation to be executed, among operations included in the neural network, is an operation executable by the accelerator or an operation executable by the host processor; and when it is detected that the operation to be executed is an operation executable by the host processor, convert the operation to be executed into an operation executable by the accelerator for execution on the accelerator.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0080219, filed on June 30, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0002] The following description relates to methods, accelerators, and electronic devices with tensor processing. Background Art

[0003] Independent hardware can be dedicated to artificial intelligence (AI). For example, AI can perform inference and learning through predetermined operations. In this way, various devices can be dedicated hardware for implementing and executing AI.

[0004] Dedicated hardware for AI can be implemented, for example, by graphics processing units (GPUs), or by repurposed field programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs). Summary of the Invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, an electronic device includes: a host processor configured to receive a request for an accelerator to process a neural network, generate an instruction executable by the accelerator in response to the request, and send the instruction to the accelerator; and an accelerator configured to execute the neural network in response to the instruction and send a result of executing the neural network to the host processor, wherein the accelerator is configured to determine an intermediate tensor corresponding to a result of a portion of an operation of a normalization layer by performing a convolution based on an input tensor and a kernel in a channel-axis direction, wherein the input tensor is of the layer and includes a plurality of channels, the number of input channels of the kernel is determined based on the input tensor, and scaling values of elements of the kernel are determined based on the number of input channels.

[0007] In one general aspect, a method for processing a neural network includes: receiving, by a host processor, a request for an accelerator to process the neural network; generating, by the host processor, an instruction executable by the accelerator in response to the request, and sending the instruction to the accelerator; and executing, by the accelerator, the neural network in response to the instruction, and sending a result of executing the neural network to the host processor, wherein the step of executing the neural network includes: determining an intermediate tensor corresponding to a result of a portion of an operation performed by a layer of the neural network corresponding to the operation to be performed by performing a convolution based on an input tensor and a kernel in a channel-axis direction, wherein the input tensor is the layer and includes a plurality of channels, the number of input channels of the kernel is determined based on the input tensor, and scaling values of elements of the kernel are determined based on the number of input channels.

[0008] In one general aspect, an electronic device for recognizing an image includes: a host processor configured to: receive a request for recognizing an image to be recognized, generate an instruction executable by an accelerator in response to the request, and send the instruction to the accelerator; and an accelerator configured to: execute a neural network with the image to be recognized as input in response to the instruction, and send a result of executing the neural network as a recognition result of the image to be recognized to the host processor, wherein the accelerator is configured to: determine an intermediate tensor corresponding to a result of a portion of an operation performed by a layer of the neural network corresponding to the operation to be performed by performing a convolution based on an input tensor and a kernel in a channel axis direction, wherein the input tensor is the layer and includes multiple channels, the number of input channels of the kernel is determined based on the input tensor and the scaling value of the elements of the kernel is determined based on the number of input channels.

[0009] In one general aspect, an electronic device includes: a host processor configured to receive a request for an accelerator to process a neural network, generate an instruction executable by the accelerator in response to the request, and send the instruction to the accelerator; and an accelerator configured to execute the neural network in response to the instruction, and send a result of executing the neural network to the host processor, wherein the accelerator is configured to determine a kernel based on information received from an on-chip memory of the accelerator indicating the number of channels to be convolved and indicating whether an operation of a layer of the neural network corresponding to the operation to be performed is an average operation or an average subtraction operation, and determine an intermediate tensor corresponding to a result of the operation of the layer by performing a convolution based on the received input tensor and the determined kernel based on an input tensor of the layer received from the on-chip memory.

[0010] In one general aspect, an electronic device includes: a host processor configured to: receive a request for an accelerator to process a neural network for a data inference result, generate an instruction executable by the accelerator in response to the request, and send the instruction to the accelerator; and an accelerator configured to: execute the neural network in response to the instruction, and send a result of executing the neural network to the host processor as a data inference result, wherein the accelerator is configured to: use a hidden layer of the neural network to determine an input tensor of a layer included in the neural network corresponding to an operation to be performed; determine an intermediate tensor corresponding to a result of the operation of the layer by performing a convolution based on the input tensor and a kernel, the number of input channels of the kernel being determined based on the input tensor, and scaling values of elements of the kernel being determined based on the number of input channels; and execute the neural network based on executing the operation to obtain a data inference result. In one general aspect, a processor-implemented tensor processing includes: receiving a request for an accelerator to process a neural network including a normalization layer; and generating instructions executable by the accelerator in response to the request, wherein, by executing the instructions, the accelerator is configured to: determine an intermediate tensor corresponding to a result of a portion of an operation of the normalization layer by performing a convolution based on an input tensor and a kernel in a channel-axis direction, wherein the input tensor is the normalization layer and includes multiple channels, the number of input channels of the kernel is determined based on the input tensor, and scaling values of elements of the kernel are determined based on the number of input channels.

[0011] The intermediate tensor may be determined by subtracting an average value of one or more elements of the input tensor from a value of each of the one or more elements through convolution, and an output tensor corresponding to an output of the normalization layer may be determined based on the intermediate tensor.

[0012] The number of input channels and the number of output channels of the kernel may be equal to the number of channels of the input tensor, and the diagonal elements of the kernel may have different scaling values than the scaling values of the remaining elements of the kernel.

[0013] The number of input channels and the number of output channels of the kernel may be equal to the number of elements included in the same channel of the input tensor, and the diagonal elements of the kernel may have scaling values different from scaling values of the remaining elements of the kernel.

[0014] Convolution can be performed between the kernel and a transformed input tensor transformed so that elements included in the same channel of the input tensor are arranged along the channel axis direction, and the intermediate tensor can be determined by converting the elements determined as a result of the convolution into the same form as the input tensor.

[0015] The step of transforming the determined elements may include transforming the determined elements so that the number of channels of the intermediate tensor is equal to the number of channels of the input tensor.

[0016] The number of input channels of the kernel may be equal to the number of channels of the input tensor, and the scaling value of the elements of the kernel may correspond to the inverse of the number of channels of the input tensor.

[0017] The intermediate tensor may be determined based on an average value of elements included in the input tensor.

[0018] Elements of the intermediate tensor may be determined separately by averaging elements included in any of: the same column in each channel of the input tensor; and the same channel of the input tensor.

[0019] The number of input channels of the kernel may be equal to the number of elements included in the same channel of the input tensor, and the scaling value of the element of the kernel may correspond to the inverse of the number of elements included in the same channel.

[0020] Convolutions may be performed in an accelerator such that input tensors are not sent outside the accelerator for performing operations according to the normalization layer.

[0021] The accelerator may be included in either or both of a user terminal configured to receive data to be inferred using a neural network and a server configured to receive data to be inferred from the user terminal.

[0022] The step of receiving the request may include receiving the request via a host processor, and the step of generating the instruction may include generating the instruction via the host processor.

[0023] A non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, configure the one or more processors to perform the method.

[0024] In another general aspect, an accelerator includes one or more processors configured to: obtain an input tensor of a normalization layer included in a neural network; obtain a kernel having a number of input channels determined based on the input tensor and including elements with scaling values determined based on the number of input channels; and determine an intermediate tensor corresponding to a result of a portion of an operation of the normalization layer by performing a convolution based on the input tensor and the kernel in a channel-axis direction.

[0025] The one or more processors may be configured to determine an intermediate tensor by subtracting an average of the one or more elements from a value of each of the one or more elements of the input tensor via convolution, and an output tensor corresponding to an output of the normalization layer may be determined based on the intermediate tensor.

[0026] The number of input channels and the number of output channels of the kernel may be equal to the number of channels of the input tensor, and the diagonal elements of the kernel may have different scaling values than the scaling values of the remaining elements of the kernel.

[0027] The number of input channels and the number of output channels of the kernel may be equal to the number of elements included in the same channel of the input tensor, and the diagonal elements of the kernel may have scaling values different from scaling values of the remaining elements of the kernel.

[0028] The one or more processors may be configured to: perform a convolution between the kernel and a transformed input tensor transformed so that elements included in the same channel of the input tensor are arranged along a channel axis direction, and determine an intermediate tensor by converting elements determined as a result of the convolution into the same form as the input tensor.

[0029] The number of input channels of the kernel may be equal to the number of channels of the input tensor, and the scaling value of the elements of the kernel may correspond to the inverse of the number of channels of the input tensor.

[0030] The number of input channels of the kernel may be equal to the number of elements included in the same channel of the input tensor, and the scaling value of the element of the kernel may correspond to the inverse of the number of elements included in the same channel.

[0031] Convolution may be performed within the accelerator such that input tensors are not sent outside the accelerator for performing operations according to the normalization layer.

[0032] An electronic device may include: a host processor configured to generate an instruction; and an accelerator, wherein the accelerator is configured to determine an intermediate tensor by executing the instruction.

[0033] In another general aspect, an electronic device includes: a host processor configured to: in response to a request for an accelerator to process a neural network including a normalization layer, generate an instruction executable by the accelerator; and the accelerator configured to: determine, by executing the instruction, an intermediate tensor corresponding to a result of a portion of an operation of the normalization layer by performing a convolution based on an input tensor and a kernel in a channel-axis direction, wherein the input tensor is of the normalization layer and includes a plurality of channels, the number of input channels of the kernel is determined based on the input tensor, and scaling values of elements of the kernel are determined based on the number of input channels.

[0034] The accelerator may be configured to determine an intermediate tensor by subtracting an average value of one or more elements of an input tensor from a value of each of the one or more elements through convolution, and an output tensor corresponding to an output of the normalization layer is determined based on the intermediate tensor.

[0035] In another general aspect, an accelerator includes: an on-chip memory, and one or more processors configured to: determine a kernel based on information received from the on-chip memory indicating the number of channels to be convolved and indicating whether an operation of a normalization layer of a neural network is an averaging operation or an average subtraction operation, and determine an intermediate tensor corresponding to a result of the operation of the normalization layer based on an input tensor of the normalization layer received from the on-chip memory by performing a convolution based on the received input tensor and the determined kernel.

[0036] In another general aspect, an electronic device includes: a host processor configured to: generate instructions in response to a request for a data inference result; and an accelerator configured to: determine, by executing the instructions: an input tensor of a normalization layer included in the neural network using a hidden layer of the neural network; determine an intermediate tensor corresponding to a result of an operation of the normalization layer by performing a convolution based on the input tensor and a kernel, the number of input channels of the kernel being determined based on the input tensor, and scaling values of elements of the kernel being determined based on the number of input channels; and determine the data inference result based on performing the operation.

[0037] The data inference result can be any of speech recognition, machine translation, machine interpretation, object recognition, pattern recognition, and computer vision.

[0038] The device is a user terminal including any one of a smart phone, a tablet computer, a laptop computer, a personal computer, a smart watch, smart glasses, a smart speaker, a smart TV, a smart refrigerator, a smart car, a smart kiosk, and an Internet of Things (IoT) device.

[0039] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 An example of an electronic device is shown.

[0041] Figure 2 and Figure 3 An example showing tensors processed by the accelerator for the first normalization layer.

[0042] Figure 4 and Figure 5 An example showing tensors for the second normalization layer being processed by the accelerator.

[0043] Figure 6 Shows an example of a kernel generator.

[0044] Figure 7 Shows an example of a neural network including a normalization layer.

[0045] Figure 8Shows an example of a normalization layer.

[0046] Figure 9 and Figure 10 An example of a tensor processing method is shown.

[0047] Figure 11 and Figure 12 An example of an electronic device is shown.

[0048] Throughout the drawings and detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0049] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the application. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the application, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, descriptions of features known in the art after understanding the disclosure of the application may be omitted.

[0050] Although the terms "first" or "second" are used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are used only to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, without departing from the teachings of the examples described herein, what is referred to as a first member, first component, first region, first layer, or first portion may also be referred to as a second member, second component, second region, second layer, or second portion.

[0051] Throughout this disclosure, when an element (such as a layer, region or substrate) is described as being "on" another element, "connected to" or "bound to" another element, the element may be directly "on" the other element, directly "connected to" or directly "bound to" the other element, or one or more other elements may be present between them. Conversely, when an element is described as being "directly on" another element, "directly connected to" or "directly bound to" another element, there may be no other elements present between them. Similarly, expressions such as "between..." and "immediately between..." and "adjacent to..." and "immediately adjacent to..." may also be interpreted as described above. The terms used herein are intended to describe specific examples only and should not limit the disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. As used herein, the term "and / or" includes any one of the relevant listed items and any combination of any two or more. As used herein, the terms "include," "comprising," and "having" indicate the presence of stated features, quantities, operations, elements, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, elements, components, and / or combinations thereof. The use of the term "may" herein with respect to an example or embodiment (e.g., for what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such features, but all examples are not limited thereto.

[0052] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs, consistent with an understanding of the present disclosure, and consistent with an understanding of the present disclosure. Unless expressly defined as such herein, terms (such as those defined in common dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense.

[0053] Hereinafter, examples will be described in detail with reference to the accompanying drawings. The specific structural or functional descriptions below are illustrative and are intended to describe only examples, and the scope of the examples is not limited to the descriptions provided in this disclosure. Those skilled in the art may make various changes and modifications based on their understanding of the disclosure of this application. The same reference numerals in the drawings represent the same elements, and known functions or configurations will be omitted herein.

[0054] Figure 1 An example of an electronic device is shown.

[0055] Reference Figure 1, the electronic device 100 may include a host processor 110 (e.g., one or more processors), an off-chip memory 120, a memory controller 130, and an accelerator 140. The host processor 110, the off-chip memory 120, the memory controller 130, and the accelerator 140 may communicate with each other via a bus.

[0056] The host processor 110 may be a device that controls the operation of components included in the electronic device 100 and may include, for example, a central processing unit (CPU). The host processor 110 may receive a request to process a neural network in the accelerator 140 and, in response to the request, generate an instruction that can be executed by the accelerator 140. The request may be for data inference based on the neural network, and in response to the generated instruction, the accelerator 140 may execute the neural network to obtain (e.g., determine) a data inference result for speech recognition, machine translation, machine interpretation, object recognition, pattern recognition, computer vision, etc. The host processor 110 may send inference target data and parameters of the neural network to the accelerator 140.

[0057] The off-chip memory 120 may be a memory provided outside the accelerator 140, and may be, for example, a dynamic random access memory (DRAM) used as a main memory of the electronic device 100. The off-chip memory 120 may be accessed through the memory controller 130. The off-chip memory 120 may store parameters of a neural network to be executed by the accelerator 140, and may be utilized when an on-chip memory in the accelerator 140 is insufficient (e.g., in terms of memory capacity) to execute the neural network by the accelerator 140.

[0058] The off-chip memory 120 may have a larger memory capacity than the on-chip memory in the accelerator 140. However, when the neural network is executed, the memory access cost of the accelerator 140 accessing the off-chip memory 120 may be greater than the memory access cost of the accelerator 140 accessing the internal on-chip memory. The memory access cost may be the power and / or time required to access the corresponding memory and read or write data.

[0059] The accelerator 140 may be an AI accelerator that infers input data by executing a neural network according to an instruction of the host processor 110, and may be a separate processor from the host processor 110. For example, the accelerator 140 may be a neural processing unit (NPU) (or neural processor), a GPU, or a tensor processing unit (TPU).

[0060] Due to the unique nature of neural network operations, accelerator 140 may be a separate, dedicated processor that can handle tasks more efficiently than general-purpose host processor 110. In this example, on-chip memory and one or more processing elements (PEs) included in accelerator 140 can be utilized by accelerator 140 when executing the neural network. The on-chip memory may be a global buffer included in accelerator 140 and may be distinguished from off-chip memory 120 provided external to accelerator 140. For example, the on-chip memory may be a scratchpad memory, static random access memory (SRAM), or the like accessible through an address space. In one embodiment, accelerator 140 may detect whether a pending operation, among operations included in the neural network, is executable by accelerator 140 or by host processor 110. If it is detected that the pending operation is executable by host processor 110, accelerator 140 may convert the pending operation into an operation executable by accelerator 140 and execute it on accelerator 140, rather than executing the operation on host processor 110.

[0061] A neural network may include multiple layers. A neural network may include an input layer, multiple hidden layers, and an output layer. Each of the multiple layers may include multiple nodes, also known as artificial neurons. Although a node may be referred to as an "artificial neuron," such a designation is not intended to convey any correlation with how a neural network architecture maps computationally or intuitively identifies information with how human neurons operate. In other words, the term "artificial neuron" is simply a term for a node in a hardware implementation of a neural network. Each node may be a computing unit with an output and one or more inputs, and the nodes may be connected to each other. Weights may be set for the connections between nodes, and the weights may be adjusted or changed. Weights may amplify, reduce, or maintain relevant data values, thereby determining the extent to which the data values influence the final result. Weighted inputs from nodes included in the previous layer may be input to each node included in the output layer. The process of inputting weighted data from a predetermined layer to the next layer may be referred to as propagation.

[0062] In order to improve the performance of a neural network, normalization can be applied to data transmitted between multiple layers of the neural network. In other words, the data distribution can be normalized by a series of operations such as obtaining the average value of the data (e.g., output data of one layer) and subtracting the value of each element of the corresponding data from the average value (e.g., to obtain data to be transmitted as input data to the next layer). Normalization applied to a neural network may include, for example, layer normalization and instance normalization. Layer normalization can be a technique for performing normalization in the channel direction regardless of the batch size, and can be applied to models in the language field such as speech recognition, machine translation, and machine interpretation. Instance normalization can be a technique for normalizing the entire image in units of channels, and can be applied to, for example, a generative adversarial network (GAN).

[0063] In addition, when an attention technique in which the size of the pooling window is equal to the size of the entire image is applied to a neural network for image processing, global average pooling may be performed, and the neural network to which global average pooling is applied may include, for example, a spatial pyramid pooling network (SPPNet) and / or a squeeze and excitation network (SENet).

[0064] As described above, each time data is transferred between layers, the neural network operation may include calculating the average of the output data of a layer and / or subtracting the average from the value of each element in the data to obtain data to be transferred to the next layer. For such averaging and / or average subtraction operations, a typical electronic device may transfer the data to be normalized from the accelerator to a specific operator (such as a host processor outside the accelerator), and the average calculated by the specific operator may be received again by the accelerator. Such data movement increases the latency and power consumption of the typical electronic device. Therefore, in order to improve the performance of the electronic device 100 relative to such a typical electronic device, the electronic device 100 of one or more embodiments may minimize data movement outside the accelerator 140 by using the neural network to perform averaging and / or average subtraction operations within the accelerator 140. For example, in one example, when the operation to be performed belongs to the operation of the normalization layer, the accelerator 140 may detect the operation to be performed as an operation that can be performed by the host processor 110 and convert the operation to be performed into an operation that can be performed by the accelerator 140 for execution on the accelerator 140.

[0065] For example, in order to implement an average operation and / or an average subtraction operation that may not be directly supported by the accelerator 140, the average operation and / or the average subtraction operation can be replaced by a convolution operation performed by the accelerator 140. The kernel for the convolution operation can be generated based on the data to be normalized. In this way, even when the convolution operation does not completely match the operation specified by the normalization layer, the accelerator 140 can obtain output data that is the same as the output generated by directly executing the operation specified by the normalization layer based on the convolution operation. In short, even without sending the input data to a specific operator (such as the host processor 110) and without modifying the hardware architecture of the accelerator 140, the accelerator 140 can obtain output data equivalent to the output of the normalization layer.

[0066] Hereinafter, a detailed description will be provided with reference to the following drawings.

[0067] Figure 2 and Figure 3 An example of processing tensors for a first normalization layer by an accelerator (eg, accelerator 140 ) is shown.

[0068] Reference Figure 2 , shows an example of a convolution operation for obtaining an average operation result according to a first normalization layer. The first normalization layer may be based on layer normalization, wherein convolution may be performed to reduce the size of a tensor in the channel axis direction. For ease of description, Figure 2 The examples of tensor shapes, kernel shapes, and element values shown in are provided, and various shapes and element values may be applied without limitation.

[0069] Figure 2 The input tensor of can be the data input to the first normalization layer. Figure 2 In the example, the input tensor has a shape of (1,3,1,4), and the value of each element can be represented by any one of a to l. Figure 2 In the example, Figure 2 The tensor structure of is represented as (batch, number of channels, height, width). In one example, the input tensor may correspond to the output data of a layer (e.g., a hidden layer) of a neural network.

[0070] Figure 2 The kernel shown in may have a number (e.g., a total number) of input channels determined based on the input tensor. For example, the number of input channels of the kernel may be equal to the number of channels of the input tensor. The number of output channels of the kernel applied to the convolution operation for obtaining the average operation result may be "1". Figure 2 In the example of , the kernel has a shape of (1,1,3,1), where the kernel shape is represented as (height, width, number of input channels, number of output channels). In addition, the scaling value of the elements included in the kernel may correspond to the inverse of the number of channels of the input tensor. Figure 2 In the example of , the scaling value is 1 / 3. In the present disclosure, for ease of description, the scaling value may also be referred to as a weight value.

[0071] The convolution between the input tensor and the kernel can be performed along the channel axis. Figure 2 In , the channel axis direction can be indicated by C and the convolution can be represented by *. For example, the values a, e, and i of the first element in each channel of the input tensor can be multiplied by the corresponding scaling value 1 / 3 of the kernel and the multiplication results can be summed so that Figure 3 The value of the first element in the average tensor shown in can be determined as (a+e+i) / 3. Similarly, convolution can also be performed on the remaining elements, so that the values of the second element, the third element, and the fourth element in the average tensor can be determined as (b+f+j) / 3, (c+g+k) / 3, and (d+h+l) / 3, respectively. Therefore, each element in the average tensor may include the average value of the corresponding element in the input tensor. The average tensor may have a shape of (1,1,1,4) that is reduced in size from the shape of (1,3,1,4) of the input tensor in the channel axis direction while maintaining the batch, height, and width sizes. In other words, the number of channels can be reduced by the convolution operation without reducing the number of batches, height, and width. By referring to the above Figure 2 By determining the average tensor as described, the average of the input tensors may be determined in the accelerator 140 through a single convolution operation without sending the input tensors to a specific operator (such as the host processor 110).

[0072] Reference Figure 3 , shows an example of a convolution operation for obtaining an average subtraction operation result according to a first normalization layer. The first normalization layer may be based on layer normalization, wherein the convolution operation may be performed to reduce the size of the tensor in the channel axis direction, which will be described in further detail below. Figure 3 The tensor shapes, kernel shapes, and element values shown in are examples for ease of description, and various shapes and element values can be applied without limitation.

[0073] Figure 3 The input tensor shown in can be the data input to the first normalization layer. Figure 3 In the example, the input tensor has a shape of (1,3,1,4), where Figure 3 The tensor structure of is represented as (batch, number of channels, height, width), and the value of each element can be represented by any one of a to l. In one example, Figure 3 The input tensor of may correspond to the output data of a layer (e.g., a hidden layer) of a neural network.

[0074] Figure 3 The kernel shown in may have a number of input channels determined based on the input tensor and a number of output channels determined based on the input tensor. For example, the number of input channels and the number of output channels of the kernel may be equal to the number of channels of the input tensor. Figure 3 In the example of , the kernel can have a shape of (1,1,3,3), where the kernel shape is expressed as (height, width, number of input channels, number of output channels). In addition, the diagonal elements and the remaining elements in the kernel can have different scaling values. For example, the diagonal elements can have a scaling value of 1-1 / n (where n represents the number of channels of the input tensor), and the remaining elements can have a scaling value of -1 / n. When Figure 3 In the example of , when the number n of channels of the input tensor is '3', the scaling values of the diagonal elements may be 2 / 3, and the scaling values of the remaining elements may be -1 / 3.

[0075] The convolution between the input tensor and the kernel can be performed in the channel axis direction to determine the average subtraction tensor. First, the scaling values 2 / 3, -1 / 3, and -1 / 3 included in the first column of the kernel can be convolved with the input tensor, whereby the value of the element in the first channel of the average subtraction tensor can be determined. For example, the values a, e, and i of the first element in each channel of the input tensor can be multiplied by the corresponding scaling values 2 / 3, -1 / 3, and -1 / 3 of the kernel, respectively, and the results of the multiplication can be summed, so that the value of the first element of the first channel of the average subtraction tensor can be determined as 2 / 3a-1 / 3e-1 / 3i (i.e., a-(a+e+i) / 3). Here, (a+e+i) / 3 corresponds to the average value of the values a, e, and i of the first element in each channel of the input tensor. Similarly, the values b, f, and j of the second element in each channel of the input tensor may be multiplied by the corresponding scaling values 2 / 3, -1 / 3, and -1 / 3 of the kernel, respectively, and the results of the multiplications may be summed, such that the value of the second element of the first channel of the average subtraction tensor may be determined to be b-(b+f+j) / 3. Here, (b+f+j) / 3 corresponds to the average of the values b, f, and j of the second element in each channel of the input tensor. Similarly, the values of the remaining elements of the first channel in the average subtraction tensor may be similarly determined, such that the values of the third and fourth elements of the average subtraction tensor may be determined to be, for example, c-(c+g+k) / 3 and d-(d+h+l) / 3, respectively.

[0076] In addition, the scaling values -1 / 3, 2 / 3, and -1 / 3 included in the second column of the kernel can be convolved with the input tensor, thereby determining the value of the element in the second channel of the average subtraction tensor. For example, the values a, e, and i of the first element in each channel of the input tensor can be multiplied by the corresponding scaling values -1 / 3, 2 / 3, and -1 / 3 of the kernel, respectively, and the results of the multiplication can be summed, so that the value of the first element of the second channel in the average subtraction tensor can be determined as e-(a+e+i) / 3. Similarly, the remaining element values of the second channel in the average subtraction tensor can be determined. In addition, the scaling values -1 / 3, -1 / 3, and 2 / 3 included in the third column of the kernel can be convolved with the input tensor, thereby determining the value of the element in the third channel of the average subtraction tensor.

[0077] Thus, each element in the average subtraction tensor may include a value obtained by subtracting the average of the values of the elements in the channel of the input tensor from the value of one of the corresponding elements in the channel of the input tensor. The average subtraction tensor may have a shape of (1, 3, 1, 4) which is the same as the shape of the input tensor. By referring to Figure 3 The determination of the average subtraction tensor and the average subtraction operation of the input tensor may be performed in the accelerator 140 through a single convolution operation without sending the input tensor to a specific operator (such as the host processor 110 ).

[0078] It can be seen that the elements in each channel of the average subtraction tensor generated by each convolution have a reduced number of channels when compared to the number of channels of the input tensor. This is because the convolution operation can be performed to reduce the size of the input tensor in the channel axis direction.

[0079] Since the remaining operations according to the normalization layer can be additionally applied to the above referenced Figure 2 and Figure 3 The convolution obtains the average tensor and the average subtraction tensor, so Figure 2 and Figure 3 The average tensor and the average subtraction tensor can also be called intermediate tensors.

[0080] Figure 4 and Figure 5 An example of processing tensors for a second normalization layer by an accelerator (eg, accelerator 140 ) is shown.

[0081] Reference Figure 4 , shows an example of a convolution operation for obtaining an average operation result according to a second normalization layer. The second normalization layer may be based on instance normalization, which normalizes the entire input tensor (e.g., an input image or a convolution feature map determined based on the input image) in units of channels, wherein instance normalization may be accomplished by changing the shape of the input tensor and then performing a convolution operation that reduces the size of the input tensor in the channel axis direction, as will be further described below. Figure 4 The tensor shapes, kernel shapes, and element values shown in are examples for ease of description, and various shapes and element values can be applied without limitation.

[0082] Figure 4 The input tensor of can be the data input to the second normalization layer. Figure 4 In the example, the input tensor has a shape of (1,2,2,3), where Figure 4 The tensor structure of is represented as (batch, number of channels, height, width), and the value of each element can be represented by any one of a to l.

[0083] To perform instance normalization based on a convolution operation, the input tensor can be converted to a predetermined shape. First, the two-dimensional elements in each channel of the input tensor can be reshaped into one-dimensional elements to generate Figure 4 The reshaped input tensor can be transposed to generate Figure 4The transposed input tensor can have a shape of (1, 6, 1, 2) and can be convolved with the kernel as described below.

[0084] Figure 4 The kernel shown in may have a number of input channels determined based on the input tensor. For example, the number of input channels of the kernel may be equal to the number of channels of the transposed input tensor. The number of output channels of the kernel applied to the convolution operation for obtaining the average operation result may be "1". Figure 4 In the example of , the kernel may have a shape of (1, 1, 6, 1), where the kernel shape is represented as (height, width, number of input channels, number of output channels). In addition, the scaling value of the elements included in the kernel may correspond to the inverse of the number of channels of the transposed input tensor. Figure 4 In the example above, the zoom value is 1 / 6.

[0085] The convolution between the input tensor and the kernel can be performed along the channel axis. Figure 4 In , the channel axis direction may be indicated by C. For example, the values a to f of the first element in each channel of the input tensor may be multiplied by the corresponding scaling value 1 / 6 of the kernel, and the results of the multiplications may be summed such that Figure 4 The value of the first element in the initial tensor of can be determined to be (a+b+c+d+e+f) / 6. Similarly, convolution can also be performed on the remaining elements similarly, so that the value of the second element in the initial tensor can be determined to be (g+h+i+j+k+l) / 6. The initial tensor can have a shape of (1,1,1,2) that is reduced in size in the channel axis direction from the shape of (1,6,1,2) of the transposed input tensor while maintaining the size of batch, height, and width. In other words, the number of channels can be reduced by the convolution operation without reducing the number of batch, height, and width.

[0086] By converting the initial tensor to the same form as the input tensor (e.g., same number of channels), Figure 4 The average tensor shown in can be determined. For example, by transposing the initial tensor, an average tensor having a shape of (1,2,1,1) can be determined. Each element in the average tensor may include the average value of the elements included in each channel of the input tensor. By referring to Figure 4 In determining the average tensor, the average value of the input tensor may be determined in the accelerator 140 through a single convolution operation without sending the input tensor to a specific operator (such as the host processor 110 ).

[0087] Reference Figure 5, shows an example of a convolution operation for obtaining an average subtraction operation result according to a second normalization layer. The second normalization layer may be based on instance normalization, where instance normalization may be performed by changing the shape of an input tensor and then performing a convolution operation that reduces the size of the input tensor in the channel axis direction. Figure 5 The tensor shapes, kernel shapes, and element values shown in are examples for ease of description, and various shapes and element values can be applied without limitation.

[0088] Figure 5 The transposed input tensor shown in may correspond to Figure 4 Therefore, the description provided above about generating a transposed input tensor for a convolution operation to obtain the result of the average subtraction operation according to the second normalization layer can be applied to Figure 5 The transposed input tensor.

[0089] Figure 5 The kernel shown in can have a number of input channels determined based on the transposed input tensor and a number of output channels determined based on the input tensor. For example, the number of input channels and the number of output channels can be equal to the number of channels of the transposed input tensor. Figure 5 In the example of , the kernel may have a shape of (1, 1, 6, 6). In addition, the diagonal elements and the remaining elements in the kernel may have different scaling values. For example, the diagonal elements may have a scaling value of 1-1 / n (where n represents the number of channels of the transposed input tensor), and the remaining elements may have a scaling value of -1 / n. Figure 5 In the example of , when the number of channels of the transposed input tensor is '6', scaling values of diagonal elements may be 5 / 6, and scaling values of remaining elements may be -1 / 6.

[0090] The convolution between the transposed input tensor and the kernel can be performed in the channel axis direction to determine the initial tensor. For example, the value of the first element in each channel of the transposed input tensor can be multiplied by the corresponding scaling value in the first column of the kernel, and the multiplication results can be summed so that the value of the first element A of the first channel of the initial tensor can be a-(a+b+c+d+e+f) / 6. Similarly, the value of the second element in each channel of the transposed input tensor can be multiplied by the corresponding scaling value in the first column of the kernel, and the multiplication results can be summed so that the value of the second element G of the first channel of the initial tensor can be g-(g+h+i+j+k+l) / 6. Similarly, the values of the remaining elements of the initial tensor can be similarly determined. Refer to the above Figure 3 The description of the convolution between the provided input tensor and the kernel can be applied to Figure 5The convolution operation between the transposed input tensor and the kernel. The initial tensor generated as a result of the convolution can have a shape of (1, 6, 1, 2), which is the same shape as the transposed input tensor.

[0091] By transforming the initial tensor into a tensor with the same size as the input tensor (e.g. Figure 4 the same form (e.g., same number of channels) as the input tensors, Figure 5 The average subtraction tensor shown in can be determined. For example, Figure 5 The transposed initial tensor having a shape of (1, 2, 1, 6) can be determined by transposing the initial tensor, and the average subtraction tensor having a shape of (1, 2, 2, 3) that is the same as the shape of the input tensor can be determined by reshaping the transposed initial tensor. As described above, the average subtraction operation of the input tensor can be performed in the accelerator 140 through a single convolution operation without sending the input tensor to a specific operator (such as the host processor 110).

[0092] Since the remaining operations according to the normalization layer can be additionally applied to the Figure 4 and Figure 5 The average tensor and average subtraction tensor obtained by convolution in , so Figure 4 and Figure 5 The average tensor and the average subtraction tensor can also be called intermediate tensors.

[0093] Figure 6 Shows an example of a kernel generator.

[0094] Reference Figure 6 , showing an on-chip memory 610, a kernel generator 620, and a PE 630. In one example, any one or any combination of the on-chip memory 610, the kernel generator 620, and the PE 630 may be included in Figure 1 accelerator 140.

[0095] The kernel generator 620 may be a kernel generator for generating a convolution (e.g., Figures 1 to 5 Any one or more of the convolutions described above) (e.g., Figures 1 to 5 The kernel generator 620 may receive the number of channels n to be convolved and a flag. The number of channels n may be determined by inputting a tensor (e.g., Figures 1 to 5 ) is determined, and may correspond to the number of channels of the input tensor, for example, in the case of layer normalization, and may correspond to the transposed input tensor, for example, in the case of instance normalization (e.g., as described above with reference to Figures 1 to 5The flag may be an indicator indicating whether the normalization operation to be replaced by convolution is an averaging operation or an average subtraction operation. For example, a flag with a value of "0" may indicate that the averaging operation is to be implemented as a convolution, and a flag with a value of "1" may indicate that the average subtraction operation is to be implemented as a convolution. If the flag has a value of "0", the kernel generator 620 may generate a kernel of the shape (1, 1, n, 1), and if the flag has a value of "1", the kernel generator 620 may generate a kernel of the shape (1, 1, n, n).

[0096] The kernel generated by the kernel generator 620 may be transferred to the PE 630, and the PE 630 may perform a convolution between the kernel and the input tensor received from the on-chip memory 610, thereby determining an intermediate tensor (e.g., as described above with reference to Figures 1 to 5 The determined intermediate tensor may be sent to the on-chip memory 610 for the remaining operations according to the normalization layer. Optionally, the remaining operations according to the normalization layer may then be performed in the PE 630, and the determined resulting output tensor may be sent to the on-chip memory 610.

[0097] Figure 7 Shows an example of a neural network including a normalization layer.

[0098] Reference Figure 7 , showing a bidirectional encoder representations from transformer (BERT) model. The normalization layer 710 is present in multiple layers included in the BERT model, wherein the normalization layer 710 is executed a total of 12 times when the corresponding model is repeated 12 times. When the normalization layers 710 are provided separately from each other, a typical electronic device implementing the BERT model is not configured to process two normalization layers 710 simultaneously outside an accelerator (e.g., in a particular operator (such as, a host processor)), and / or the cost of data movement (e.g., memory access cost) may be significant each time the normalization layer 710 is processed outside the accelerator. However, when using the above reference Figures 1 to 6 When the convolution described above replaces the normalization layer 710, the operation of the normalization layer 710 can be performed in the accelerator 140 of one or more embodiments, and such costs can be effectively avoided. Figure 7 In the example of , an attention mask (attention_mask) having a shape of (1, 128, 128) represented as (minbatch, length, extent) and another input having a shape of (128, 768) represented as (length, hidden layer size) may be input to the attention layer.

[0099] Figure 8 Shows an example of a normalization layer.

[0100] Reference Figure 8 , showing the internal structure of the normalization layer. Figure 8 The normalization layer shown in FIG can be used to perform the Figure 7 The layer normalization layer in the BERT model of . The average subtraction operation 810 can be performed in the first part of the normalization layer. By using a single convolution (e.g., Figures 1 to 7 By replacing the average subtraction tensor 810 with any one or more of the convolutions described above, the electronic device 100 of one or more embodiments can effectively suppress the movement of the input tensor outside the accelerator 140, thereby reducing the memory access cost of the accelerator 140 accessing the off-chip memory 120.

[0101] Figure 9 and Figure 10 An example of a tensor processing method is shown.

[0102] Reference Figure 9 , showing the processing performed by a host processor (e.g., Figure 1 In operation 910, the host processor may receive a request for an accelerator to process a neural network (e.g., a neural network including a normalization layer). In operation 920, the host processor may generate an instruction that can be executed by the accelerator in response to the request, and send the instruction to the accelerator. In one example, the accelerator may execute the neural network in response to the instruction, and send the result of executing the neural network to the host processor. For example, the accelerator may detect whether an operation to be executed in the operation included in the neural network belongs to an operation that can be executed by the accelerator or an operation that can be executed by the host processor; when it is detected that the operation to be executed belongs to an operation that can be executed by the host processor, the accelerator may convert the operation to be executed into an operation that can be executed by the accelerator for execution on the accelerator. The accelerator may convert the operation to be executed into an operation that can be executed by the accelerator by performing the following steps on the accelerator: determining an intermediate tensor corresponding to the result of a portion of the operation executed by the layer of the neural network corresponding to the operation to be executed by performing a convolution based on the input tensor and the kernel in the channel axis direction, wherein the input tensor is the layer and has multiple channels, and the kernel has a number of input channels determined based on the input tensor and includes an element with a scaling value determined based on the number of input channels. In one example, when the instruction is executed by the accelerator, the accelerator may determine an intermediate tensor corresponding to a portion or at least a portion of the result of the operation executed by the normalization layer by performing a convolution based on the input tensor and the kernel in the channel axis direction.

[0103] Reference Figure 10 , showing the accelerator (e.g., Figure 1 In operation 1010, the accelerator may obtain an input tensor of a normalization layer included in a neural network. In operation 1020, the accelerator may obtain a kernel having a number of input channels determined based on the input tensor and including elements with scaling values determined based on the number of input channels. In operation 1030, the accelerator may determine an intermediate tensor corresponding to a portion or at least a portion of the result of the operation performed by the normalization layer by performing a convolution based on the input tensor and the kernel in the channel axis direction.

[0104] The tensor processing method can be applied to training or inference on the algorithm side, the compiler side, or hardware using multi-head attention networks (such as automatic speech recognition (ASR) or transformers). In addition, even when low-precision quantization schemes are applied to the tensor processing method, performance gains can be guaranteed while maintaining system accuracy.

[0105] Reference Figures 1 to 8 The description provided can be applied to Figures 9 and 10 The operations shown in .

[0106] Figure 11 and Figure 12 An example of an electronic device is shown.

[0107] Reference Figure 11 , electronic devices (e.g., Figure 1 The electronic device 100) can be implemented as a server 1100.

[0108] The server 1100 may be a separate device from the user terminal controlled by the user, and may communicate with the user terminal via a wired and / or wireless network. In one example, the server 1100 may be an accelerator system for recognizing images. Data to be inferred using a neural network (e.g., an image to be recognized) may be collected from the user terminal and sent to the server 1100 via the network, and the server 1100 may be configured as described above with reference to FIG. Figures 1 to 10The described tensor processing method uses accelerator 1120 to process layers included in a neural network (e.g., normalization layers). For example, host processor 1110 may receive a request to identify an image to be recognized, generate instructions executable by accelerator 1120 in response to the request, and send the instructions to accelerator 1120. In response to the instructions, accelerator 1120 may execute the neural network using the image to be recognized as input and send the results of executing the neural network to host processor 1110 as recognition results for the image to be recognized. In this example, the input tensors of the normalization layer may not be sent to host processor 1110 outside of accelerator 1120. Server 1100 may then obtain inference results (e.g., image recognition results) from the neural network and return the inference results to the user terminal. In this example, since the tensors are not sent outside of accelerator 1120, the latency and power consumption of tensor transmission can be reduced, and the efficiency of server 1100 in recognizing images can be improved. For example, user terminals may include various computing devices (such as smart phones, tablets, laptops, and personal computers), various wearable devices (such as smart watches and smart glasses), various home appliances (such as smart speakers, smart TVs, and smart refrigerators), smart cars, smart kiosks, and Internet of Things (IoT) devices.

[0109] The user terminal may simply provide the user with the inference result received from the server 1100 , or perform subsequent operations based on the inference result.

[0110] Reference Figure 12 , electronic devices (e.g., Figure 1 The electronic device 100) can be implemented as a user terminal 1200. Figure 12 , for ease of description, the user terminal 1200 is shown as a smartphone. However, any device controlled by a user may be applied hereto without limitation. The user terminal 1200 may obtain data to be inferred using a neural network directly from the user, and the accelerator 1220 may process the normalization layer included in the neural network according to the above-mentioned tensor processing method. In this example, the input tensor of the normalization layer may not be sent to the host processor 1210 outside the accelerator 1220, and the accelerator 1220 may determine the data inference result based on the steps of performing the operation according to the normalization layer. In addition, the user terminal 1200 may obtain the inference result from the neural network and simply provide the inference result to the user or perform subsequent operations based on the inference result.

[0111] In this regard Figures 1 to 12The electronic device, host processor, off-chip memory, memory controller, accelerator, on-chip memory, kernel generator, PE, server, user terminal, electronic device 100, host processor 110, off-chip memory 120, memory controller 130, accelerator 140, on-chip memory 610, kernel generator 620, PE 630, server 1100, host processor 1110, accelerator 1120, user terminal 1200, host processor 1210, accelerator 1220 and other devices, devices, units, modules and components described are implemented by hardware components or are representatives of hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component, or two or more hardware components.The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0112] Perform the operations described in this application Figures 1 to 12 The method shown in is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as described above to execute instructions or software to perform the operations performed by the method described in this application. For example, a single operation, or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation, or two or more operations.

[0113] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0114] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0115] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered as descriptive only and not for limiting purposes. The description of the features or aspects in each example should be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order and / or if the components in the described systems, architectures, devices or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, suitable results can be achieved.

Claims

1. An electronic device comprising: a host processor configured to: receive a request for the accelerator to process a neural network, generate instructions executable by the accelerator in response to the request, and send the instructions to the accelerator; and an accelerator configured to: execute the neural network in response to the instruction and send a result of executing the neural network to the host processor, The accelerator is configured to determine an intermediate tensor corresponding to a result of a portion of an operation of the normalization layer by performing a convolution based on the input tensor and the kernel in a channel axis direction, wherein the input tensor is of the layer and includes a plurality of channels, the number of input channels of the kernel is determined based on the input tensor, and the scaling values of the elements of the kernel are determined based on the number of input channels.

2. The electronic device according to claim 1, wherein The accelerator is configured to determine an intermediate tensor by subtracting a mean of the one or more elements of the input tensor from a value of each of the one or more elements via a convolution, and An output tensor corresponding to the output of the normalization layer is determined based on the intermediate tensor.

3. The electronic device according to claim 1, wherein The number of input channels and the number of output channels of the kernel are equal to the number of channels of the input tensor, and The diagonal elements of the kernel have scaling values that are different from the scaling values of the remaining elements of the kernel.

4. The electronic device according to claim 1, wherein The number of input channels and the number of output channels of the kernel are equal to the number of elements included in the same channel of the input tensor, and The diagonal elements of the kernel have scaling values that are different from the scaling values of the remaining elements of the kernel.

5. The electronic device according to claim 1, wherein The accelerator is configured as: performs a convolution between the kernel and the transformed input tensor that is transformed so that elements included in the same channel of the input tensor are arranged along the channel axis direction, and An intermediate tensor is determined by converting elements determined as a result of the convolution into the same form as the input tensor. The electronic device according to claim 5 , wherein: The process of converting the determined elements includes converting the determined elements so that the number of channels of the intermediate tensor is equal to the number of channels of the input tensor.

7. The electronic device according to claim 1, wherein The number of input channels of the kernel is equal to the number of channels of the input tensor, and The scaling value of the kernel's elements corresponds to the inverse of the number of channels of the input tensor.

8. The electronic device according to claim 1, wherein The number of input channels of the kernel is equal to the number of elements included in the same channel of the input tensor, and The scaling value of the elements of the kernel corresponds to the inverse of the number of elements included in the same channel.

9. The electronic device according to claim 1, wherein: Determines an intermediate tensor based on the average of the elements included in the input tensors.

10. The electronic device according to claim 9, wherein: The elements of the intermediate tensor are determined by averaging the elements contained in either: the same column in each channel of the input tensor; and The same channel of the input tensor.

11. The electronic device according to any one of claims 1 to 10, wherein: When the operation to be performed belongs to an operation of the normalization layer, the accelerator detects the operation to be performed as an operation that can be performed by the host processor, and wherein convolution is performed within the accelerator such that input tensors are not sent from outside the accelerator for performing operations according to the normalization layer.

12. The electronic device according to claim 1, wherein The accelerator is included in either or both of a user terminal configured to receive data to be inferred using a neural network and a server configured to receive data to be inferred from the user terminal.

13. The electronic device according to claim 1, wherein The process of receiving the request includes receiving the request by a host processor, and the process of generating the instruction includes generating the instruction by a host processor.

14. A method of processing a neural network, comprising: receiving, by the host processor, a request for the accelerator to process the neural network; generating, by the host processor, an instruction executable by the accelerator in response to the request, and sending the instruction to the accelerator; and executing the neural network in response to the instruction through the accelerator and sending the result of executing the neural network to the host processor, The step of executing the neural network includes determining an intermediate tensor corresponding to a result of a portion of an operation of a layer of the neural network corresponding to the operation to be performed by performing a convolution based on the input tensor and the kernel in the channel axis direction, wherein the input tensor is of the layer and includes a plurality of channels, the number of input channels of the kernel is determined based on the input tensor, and the scaling values of the elements of the kernel are determined based on the number of input channels.

15. The method according to claim 14, wherein The intermediate tensor is determined by subtracting the mean of the one or more elements of the input tensor from the value of each of the one or more elements via the convolution, and An output tensor corresponding to the output of the layer is determined based on the intermediate tensor.

16. The method according to claim 14, wherein The number of input channels and the number of output channels of the kernel are equal to the number of channels of the input tensor, and The diagonal elements of the kernel have scaling values that are different from the scaling values of the remaining elements of the kernel.

17. The method according to claim 14, wherein: The number of input channels and the number of output channels of the kernel are equal to the number of elements included in the same channel of the input tensor, and The diagonal elements of the kernel have scaling values that are different from the scaling values of the remaining elements of the kernel.

18. The method according to claim 14, wherein The convolution is performed between the kernel and the transformed input tensor that is transformed so that the elements included in the same channel of the input tensor are arranged along the channel axis direction, and The intermediate tensor is determined by converting the elements determined as a result of the convolution into the same form as the input tensor.

19. The method according to claim 18, wherein The step of transforming the determined elements includes transforming the determined elements so that the number of channels of the intermediate tensor is equal to the number of channels of the input tensor.

20. The method according to claim 14, wherein The number of input channels of the kernel is equal to the number of channels of the input tensor, and The scaling value of the kernel's elements corresponds to the inverse of the number of channels of the input tensor.

21. The method according to claim 14, wherein Determines an intermediate tensor based on the average of the elements included in the input tensors.

22. The method according to claim 21, wherein The elements of the intermediate tensor are determined by averaging the elements contained in either: the same column in each channel of the input tensor; and The same channel of the input tensor.

23. The method according to claim 14, wherein The number of input channels of the kernel is equal to the number of elements included in the same channel of the input tensor, and The scaling value of the elements of the kernel corresponds to the inverse of the number of elements included in the same channel.

24. The method according to any one of claims 14 to 23, wherein: When the operation to be performed belongs to an operation of the normalization layer, the accelerator detects the operation to be performed as an operation that can be performed by the host processor, and Wherein, convolution is performed in an accelerator such that input tensors are not sent from outside the accelerator for performing operations according to the normalization layer.

25. The method according to claim 14, wherein The accelerator is included in either or both of a user terminal configured to receive data to be inferred using a neural network and a server configured to receive data to be inferred from the user terminal.

26. The method according to claim 14, wherein The step of receiving the request includes receiving the request via a host processor, and the step of generating the instruction includes generating the instruction via a host processor.

27. An electronic device for recognizing an image, comprising: A host processor is configured to: receive a request for identifying an image to be identified, and generate, in response to the request, an instruction executable by the accelerator, so as to process a neural network including a normalization layer through the accelerator; and an accelerator configured to: execute a neural network with the image to be recognized as input in response to the instruction, and send a result of executing the neural network as a recognition result of the image to be recognized to the host processor, The accelerator is configured to determine an intermediate tensor corresponding to a result of a portion of an operation of a layer of the neural network corresponding to the operation to be performed by performing a convolution based on the input tensor and the kernel in a channel axis direction, wherein the input tensor is of the layer and includes a plurality of channels, the number of input channels of the kernel is determined based on the input tensor, and the scaling values of the elements of the kernel are determined based on the number of input channels.

28. The electronic device according to claim 27, wherein: The accelerator is configured to determine an intermediate tensor by subtracting a mean of one or more elements of an input tensor from a value of each of the one or more elements via a convolution, and An output tensor corresponding to the output of the layer is determined based on the intermediate tensor.

Images