Neural network processing devices and their operation methods that effectively utilize memory space

Abstract

A neural network processing device and its operating method that effectively utilize memory space are disclosed. The operating method includes: storing an input feature map in memory; generating a first output line of an output feature map by performing a convolution operation between a first input line group of the input feature map and a weight kernel; generating a first output based on the first output line, which includes a block of operations comprising the convolution operation; and storing the first output in memory by overwriting the first output to the memory space where the input feature map is stored.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0171929, filed on December 20, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to neural network processing devices for efficient use of memory space and methods of operation thereof. Background Technology

[0003] The automation of recognition processing techniques has been achieved through neural network models implemented, for example, by processors as dedicated computing architectures, which, after extensive training, provide an intuitive mapping between input and output patterns. The ability to generate such mappings through training is the learning capability of neural networks. Furthermore, specialized neural networks, trained through dedicated training, possess, for example, the ability to generalize to provide relatively accurate outputs for untrained input patterns. However, since the large amounts of input and output data used in neural network models need to be stored in memory, techniques to improve the efficiency of memory space utilization are needed. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, an operational method for a neural network processing device for efficiently utilizing memory space includes: storing an input feature map in memory; generating a first output line of an output feature map by performing a convolution operation between a first set of input lines of the input feature map and a weight kernel; generating a first output based on the first output line, comprising a block of operations including the convolution operation; and storing the first output in memory by overwriting the first output to the memory space where the input feature map is stored.

[0006] In one general aspect, a neural network processing device for efficiently utilizing memory space includes: a processor configured to: store an input feature map in memory; generate a first output line of an output feature map by performing a convolution operation between a first set of input lines of the input feature map and a weight kernel; generate a first output based on the first output line, comprising a block of operations including the convolution operation; and store the first output in memory by overwriting the first output to the memory space where the input feature map is stored.

[0007] In one general aspect, an operational method for a neural network processing device for efficiently utilizing memory space includes: storing an input feature map in memory; generating a first output line of an output feature map by performing a convolution operation between a first group of input lines of the input feature map and a weight kernel; storing the first output line in a first memory space of the memory; generating a second output line of the output feature map by performing a convolution operation between a second group of input lines of the input feature map and a weight kernel; and storing the second output line in a second memory space by overwriting the second output line into a second memory space of the memory that includes the first input lines of the first group of input lines.

[0008] In one general aspect, a method for processing convolution operations associated with layers in a neural network (the method being performed by a processor) includes: generating a first output line of an output feature map by performing a convolution operation between a first set of input lines of an input feature map and a weight kernel; generating a first output based on the first output line, comprising a block of operations including the convolution operation; and storing the first output in memory by overwriting the first output to a memory space storing the input feature map.

[0009] The storage step may include overwriting the first output into memory space of at least one input feature element in the input feature map that is no longer used for processing the operation block. The operation block may also include any one or any combination of pooling operations, skip connections, and at least one other convolution operation different from the convolution operation.

[0010] The method may further include: obtaining information related to the memory space of the input feature map, and allocating memory space for the output feature map such that the output feature map is overwritten into at least a portion of the memory space of the input feature map.

[0011] The step of generating the first output line may include: accumulating a weighted input element vector, which is the product of the input element vector of the first input line group and the weight elements of the weight kernel, to generate the first output line. The step of generating the first output line may also include: generating a first output element vector of the first output line by performing a convolution operation between the input lines of the first input line group and a first weight kernel corresponding to the first output channel, wherein the first output element vector may correspond to the first output channel.

[0012] The first input element vector corresponding to the first offset in the input element vector can be multiplied with the first weight element corresponding to the first offset in the weight element vector. If the first weight element is the zero weight element corresponding to "0", the multiplication between the first input element vector and the first weight element can be omitted.

[0013] In response to the operation block including a first operation corresponding to a convolution operation and a second operation different from the first operation and using a first output line, the step of generating a first output may include: allocating an additional buffer for the first output line, the additional buffer corresponding to the size of the first output line; performing the first operation and storing the result of the first operation as the first output line in the additional buffer; allocating an output line buffer for the first output, the output line buffer corresponding to the size of the first output; and performing the second operation using the first output line and storing the result of the second operation as the first output in the output line buffer.

[0014] The input feature map can be stored in memory as a line data structure, with data sorting applied to the line data structure in the order of width, channel, and height. The line data structure can differ from a planar data structure and from an interleaved data structure. In a planar data structure, data sorting is applied in the order of width, height, and channel; in an interleaved data structure, data sorting is applied in the order of channel, width, and height. The input feature map may include multiple input lines, each including an input feature vector. The first output line may include output feature vectors from multiple output channels.

[0015] In another general aspect, an apparatus for processing convolutional operations associated with layers in a neural network includes: a processor; and a memory configured to store instructions executable by the processor, wherein, in response to the instructions being executed by the processor, the processor is configured to: generate a first output line of an output feature map by performing a convolutional operation between a first set of input lines of an input feature map and a weight kernel; generate a first output based on the first output line, comprising a block of operations including the convolutional operation; and store the first output in the memory by overwriting the first output to a memory space storing the input feature map.

[0016] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0017] Figure 1 An example of a processing device for handling convolution operations in neural networks is shown.

[0018] Figure 2 Examples of data related to convolution operations are shown.

[0019] Figure 3 An example of a line data structure is shown.

[0020] Figure 4 This shows an example of convolution operations and the processing of storing the output line.

[0021] Figure 5 and Figure 6 An example of generating an output element vector is shown.

[0022] Figures 7 to 9 This shows an example of the storage status of the input and output lines in the memory.

[0023] Figure 10 An example is shown where additional memory space is allocated in consideration of the relationship between the size of the output feature map and the size of the input feature map.

[0024] Figure 11 An example of computational processing related to computational blocks that include skip connections is shown.

[0025] Figure 12 An example of computation processing associated with a block of operations, including pooling operations, is shown.

[0026] Figure 13 This is a flowchart illustrating an example of setting up a memory.

[0027] Figure 14 This is a flowchart illustrating an example of convolution operation processing.

[0028] Figure 15 An example of memory mapping is shown.

[0029] Figure 16 This is a flowchart illustrating an example of a convolution operation processing method.

[0030] Figure 17 An example of a processing device is shown.

[0031] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0032] The detailed structural or functional descriptions provided below are merely examples, and various alternatives and modifications may be made to the examples. Therefore, the examples are not to be construed as limiting the scope of the disclosure and should be understood to include all changes, equivalents, and alternatives within the scope of the disclosed technology.

[0033] Terms (such as first, second, etc.) may be used to describe components. Each of these terms is not used to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from one or more other components. For example, the first component may be referred to as the second component, and similarly, the second component may be referred to as the first component.

[0034] Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It will also be understood that the terms “comprising” and / or “including” as used herein indicate the presence of the stated features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0035] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0036] The examples will be described in detail below with reference to the accompanying drawings. In the drawings, the same reference numerals are used for the same elements.

[0037] Figure 1 An example of a processing device used to process convolution operations in a neural network is shown. (See reference...) Figure 1 The processing device (e.g., a neural network processing device) 100 includes a neural network 110 and processes operations associated with the neural network 110. For example, operations associated with the neural network 110 include object recognition operations and user authentication operations.

[0038] The neural network 110 performs object recognition or user verification operations by mapping input and output data in a non-linear relationship based on deep learning. Deep learning is a machine learning technique used to solve problems (such as image recognition or speech recognition) on large datasets. Deep learning is explained as an optimization problem-solving process that finds the point that minimizes energy while training the neural network 110 using prepared training data. Through supervised or unsupervised learning in deep learning, the structure of the neural network 110, or the weights corresponding to the model, is obtained, and the input and output data are mapped to each other through the weights.

[0039] The neural network 110 corresponds to a deep neural network (DNN) comprising multiple layers. The multiple layers include an input layer, at least one hidden layer, and an output layer. Figure 1 The first, second, and nth layers shown correspond to at least a portion of a plurality of layers (e.g., n is a positive integer greater than 2). The neural network 110 includes any one or any combination of fully connected networks, convolutional neural networks (CNNs), and recurrent neural networks (RNNs). For example, at least a portion of the plurality of layers in neural network 110 corresponds to a CNN, and another portion of the plurality of layers in neural network 110 corresponds to a fully connected network.

[0040] The data input to each layer in a CNN is called an input feature map, and the data output from each layer is called an output feature map. Input and output feature maps are also referred to as activation data. In the input layer, the input feature map corresponds to the input data.

[0041] To handle operations associated with neural network 110, processing device 100 processes various operational blocks. Operational blocks include at least one convolution operation (e.g., a single convolution operation or multiple convolution operations), skip connections, and any one or any combination of pooling operations. For example, operational blocks include convolution operations associated with layers (e.g., convolutional layers) in neural network 110. Processing device 100 performs convolution operations between the input feature map and the weight kernel for each convolutional layer and generates an output feature map based on the result of the convolution operation. If the width and depth of neural network 110 are sufficiently large, neural network 110 can have a capacity sufficient to achieve the intended function. When learning from a sufficiently large amount of training data through appropriate training processing, neural network 110 can achieve optimized performance.

[0042] The weight kernel is indicated as being "pre-determined". Here, "pre-determined" means before the neural network 110 is "started". A "started" neural network 110 means a neural network 110 ready to perform inference. For example, a "started" neural network 110 includes a neural network 110 loaded into memory, or a neural network 110 that receives input data for inference after being loaded into memory. For example, the weight kernel may be trained and subsequently used for inference operations.

[0043] In sliding window convolution operations, regions in the input feature map that have been scanned once are no longer used. Examples show how the characteristics of sliding window convolution operations can be leveraged to improve memory space utilization, as will be further described below. For instance, to store the output feature map, memory space in the input feature map that is no longer used can be reused. In this way, limited memory space can be used efficiently, and anticipated overflows in memory space can be prevented, thereby improving the functionality of a computer on which one or more embodiments can be implemented.

[0044] Figure 2 Examples of data related to convolution operations are shown. (See reference...) Figure 2 The output feature map 230 is generated based on the convolution operation between the weight kernel 210 and the input feature map 220 (where, Figure 2(The asterisk "*" indicates a convolution operation). Weight kernel 210, input feature map 220, and output feature map 230 are each represented as a plane (e.g., a two-dimensional matrix). For example, weight kernels 1 to D each include C weight planes, input feature map 220 includes C input planes, and output feature map 230 includes D output planes. The C weight planes and C input planes correspond to the input channels, and the D output planes correspond to the output channels. Furthermore, C corresponds to the number of input channels, and D corresponds to the number of output channels.

[0045] Each plane comprises elements of a predetermined width. For example, each weight plane has a size of K×K, and each input plane and each output plane has a size of W×H. Here, W, K, and H each represent the number of elements. The elements of the weight plane are called weight elements, the elements of the input plane are called input elements, and the elements of the output plane are called output elements. The convolution operation is performed element-wise.

[0046] For ease of description, it is assumed that the width and height of the weight plane are both K, and the size of the input plane and the output plane are both W×H. However, in some examples, the width and height of the weight plane may be different from each other, and / or the size of the input plane and the size of the output plane may be different from each other.

[0047] Figure 3 An example of a line data structure is shown. Figure 3 In this architecture, interleaved data structures and linear data structures are represented as three-dimensional (3D) blocks rather than planar representations. This difference in representation is solely for ease of description. The planar and 3D block representations are freely interchangeable. Assuming... Figure 3 The planar data structure, interleaved data structure, and linear data structure each include the same number of input elements (e.g., W×H×C input elements).

[0048] Figure 3 The planar, interleaved, and linear data structures each correspond to a data sorting scheme used to store input elements in memory. Data sorting determines the order in which memory addresses are allocated to data when storing it in memory.

[0049] Reference Figure 3In planar data structures, data sorting is applied in the order of width, height, and channel. In interleaved data structures, data sorting is applied in the order of channel, width, and height. Planar data structures exhibit good computational efficiency when the individual input planes are treated as a whole and are not suitable for processing outputs in relatively small units. Interleaved data structures are advantageous for processing outputs in relatively small units but are not suitable for effectively utilizing data sparsity. For example, utilizing data sparsity includes zero skipping.

[0050] A linear data structure, which combines the advantages of planar and interleaved data structures, is utilized. In the case of the linear data structure, data sorting is applied in the order of width, channel, and height. The linear data structure includes multiple input lines. For example, input line 310 includes multiple input feature vectors, each containing multiple input elements. Here, each input feature vector corresponds to a predetermined input channel. For example, input feature vector 311 corresponds to the first input channel, and input feature vector 312 corresponds to the second input channel.

[0051] In the case of online data structures, outputs are generated for each line as a relatively small unit (e.g., compared to the units of the input plane processed using a planar data structure). Furthermore, sparsity can be used to accelerate the process during the multiplication of the input element vector of each input line with its corresponding weight element, as will be described below.

[0052] Figure 4 This illustrates an example of convolution operations and the processing of storing the output line. (See also...) Figure 4 The operation block 401, which includes convolution operations, is processed. Operation block 401 receives a weight kernel 410 and an input feature map 420 as input, and outputs an output feature map. For ease of description, Figure 4 Instead of showing the entire output feature map, the output line Out_1 corresponding to a portion of the output feature map is shown. According to the line data structure, the input feature map 420 includes input lines In_1 to In_h, and the output feature map includes the output line containing output line Out_1. Each input line In_1 to In_h includes an input feature vector, and each output line includes an output feature vector. Each input feature vector corresponds to a different input channel, and each output feature vector corresponds to a different output channel. Let "h" be the number of both input and output lines.

[0053] To process operation block 401, the computing device performs a convolution operation between weight kernel 410 and input feature map 420. While sequentially processing input lines In_1 to In_h for each convolution unit corresponding to the height K of weight kernel 410, the computing device generates output lines by performing convolution operations. The input lines In_1 to In_h corresponding to the convolution unit are called an input line group. In one example, the number of input lines included in the input line group may correspond to the height K of weight kernel 410. For example, assuming the height K of weight kernel 410 is "3", a first input line group including three input lines In_1 to In_3 is defined.

[0054] The computing device generates the output line Out_1 by performing a convolution operation between the first input line group and the weight kernel 410. For example, the computing device generates the output element vector 11 by performing a convolution operation between the first input line group and the weight kernel 1. The weight kernel 1 and the output element vector 11 correspond to the same output channel. Furthermore, the computing device generates the output element vector 12 by performing a convolution operation between the first input line group and the weight kernel 2, and generates the output element vector 1D by performing a convolution operation between the first input line group and the weight kernel D. The weight kernel 2 and the output element vector 12 correspond to the same output channel, and the weight kernel D and the output element vector 1D correspond to the same output channel. The output element vectors 11 to 1D constitute the output line Out_1. The convolution operations are performed cumulatively. The cumulative convolution operation will be further described below.

[0055] The computing device generates the output of operation block 401 based on output line Out_1. For example, if operation block 401 includes operations other than convolution, the output of operation block 401 is generated by additionally performing the corresponding operations based on output line Out_1. Figure 4 The example assumes that operation block 401 includes a single convolution operation. Therefore, output line Out_1 corresponds to the output of operation block 401.

[0056] The computing device stores the data generated during the convolution operation between the first input line group and the weight kernel 410 in an output line buffer corresponding to the size of the output line Out_1. Then, the computing device stores the output line Out_1 in memory by overwriting it in the output line buffer to the memory space storing the input feature map 420. The computing device also overwrites the output line Out_1 to the memory space of at least one input feature element in the input feature map 420 that is no longer used to process the operation block 401. For example, if the input line In_1 is no longer used to process the operation block 401, the computing device overwrites the output line Out_1 to the memory space storing the input line In_1.

[0057] If the operations associated with the first input line group are completed, the processing device defines a second input line group including input lines In_2 to In_4 and performs a convolution operation between the second input line group and the weight kernel 410. As described above, the computing device generates an output feature map by sequentially performing convolution operations associated with each input line group. During the processing of generating the output feature map, each output line is overwritten to at least a portion of the memory space of the input feature map 420 via an output line buffer. For example, when output line Out_2 is no longer used for processing operation block 401 after being used to generate input line In_2, the computing device may overwrite output line Out_2 into the memory space storing input line In_2.

[0058] Figure 5 and Figure 6 An example of generating an output element vector is shown. The convolution operation is performed by accumulating the intermediate results of the convolution operation into the output feature map. Therefore, this eliminates the need for buffered operations that convert the weight kernels or input feature maps into a form suitable for convolution and store them in a buffer. Furthermore, Single Instruction Multiple Data (SIMD) uses a linear data structure for processing. SIMD represents a type of processor operation that processes multiple data points using a single instruction. Therefore, the speed of convolution operations is greatly improved.

[0059] Reference Figure 5 The diagram illustrates input line group 500 and weight kernel 510. Input line group 500 corresponds to one of the input line groups of the input feature map defined during convolution operation processing. For example, input line group 500 corresponds to... Figure 4 The first input line group. Weight kernel 510 corresponds to one of the weight kernels used during the convolution operation. For example, weight kernel 510 corresponds to Figure 4 Weight kernel 1. As described above, the output element vector (e.g., Figure 4 The output element vector 11) is generated based on the convolution operation between the input line group 500 and the weight kernel 510.

[0060] In the following text, the cumulative convolution operation will be described based on input element vectors 5011, 5021, and 5031. This description also applies to operations related to the remaining input element vectors 5012, 5013, 5022, 5023, 5032, and 5033. Furthermore, the following description does not restrict the order in which input element vectors 5011 to 5033 are applied during the cumulative convolution operation. That is, input element vectors 5011 to 5033 can be applied in various orders during the cumulative convolution operation. For example, to suit linear data structures, input element vectors 5011, 5012, and 5013, input element vectors 5021, 5022, and 5023, and input element vectors 5031, 5032, and 5033 are applied sequentially to the convolution operation.

[0061] Reference Figure 5 The input plane 520 is shown. The sliding region 521 of the input plane 520 corresponds to the input element vectors 5011, 5012, and 5013. Figure 5 In this example, we assume the sliding step size is "1", and zero-padding is applied to each of the horizontal and vertical directions of the input plane 520 using two rows of element vectors. Therefore, the width of the input plane 520 is "W+2", and the height of the input plane 520 is "H+2".

[0062] Convolution is performed between the input plane 520 and the weight plane 530. It is assumed that the input plane 520 and the weight plane 530 correspond to the same input channel. As the weight plane 530 slides over the input plane 520, the regions corresponding to the weight elements of the weight plane 530 are determined in the sliding region 521. More specifically, weight element w1 corresponds to region 5211, weight element w2 corresponds to region 5212, and weight element w3 corresponds to region 5213. The remaining weight elements w4 to w9 correspond to the remaining regions 5221 to 5233. In one example, region 5211 includes regions A, B, and C; region 5212 includes regions B, C, and D; region 5213 includes regions C, D, and E; region 5221 includes regions F, G, and H; region 5222 includes regions G, H, and I; region 5223 includes regions H, I, and J; region 5231 includes regions K, L, and M; region 5232 includes regions L, M, and N; and region 5233 includes regions M, N, and O.

[0063] The input element vectors are extracted from regions 5211 to 5233 and stored in registers r1 to r9. For example, the first input element vector of region 5211 is stored in register r1, and the second input element vector of region 5212 is stored in register r2. As described above, the input element vectors are stored sequentially in registers r1 to r9.

[0064] Each input element vector is multiplied element-wise with the corresponding weight element among the weight elements w1 to w9 (e.g., by symbol...). (represented by...), thus generating a weighted input element vector. A corresponding pair is determined based on the offset of each element in the input element vector and the offset of each of the weight elements w1 to w9. For example, the first input element vector corresponding to the first offset in the input element vector is multiplied by the first weight element corresponding to the first offset among the weight elements w1 to w9. The first input element vector in region 5211 is stored in register r1 and multiplied by the weight element w1, thus generating a first weighted input element vector. The second input element vector in region 5212 is stored in register r2 and multiplied by the weight element w2, thus generating a second weighted input element vector. Regions 5211 to 5233, the magnitudes of the input element vectors and the weighted input element vectors correspond to the SIMD operation units.

[0065] According to the cumulative convolution operation, each input element vector is multiplied by its corresponding weight element. If there exists a zero-weight element corresponding to "0", the operation associated with that zero-weight element can be omitted from the processing. For example, if weight element w1 is a zero-weight element corresponding to "0", the multiplication between the first input element vector and weight element w1 is omitted. Therefore, zero skipping can be efficiently handled by software according to the cumulative convolution operation.

[0066] By accumulating (e.g., summing) the weighted input element vector generated through the above processing, a cumulative vector corresponding to the sliding region 521 is generated. Furthermore, as processing is performed iteratively for each sliding region, a cumulative vector corresponding to each sliding region is generated, and these cumulative vectors are accumulated to form the output element vector. (Refer to...) Figure 6 The previously stored accumulated vector (hereinafter referred to as the first accumulated vector) is loaded from the accumulation region 611 of the output plane 610 and stored in register r10. If a new accumulated vector (hereinafter referred to as the second accumulated vector) is generated through registers r1 to r9 (e.g., by accumulating the weighted input element vector generated based on registers r1 to r9 and weight elements w1 to w9), the first accumulated vector and the second accumulated vector are accumulated in register r10 and stored in the accumulation region 611.

[0067] Figure 6 Assume that the process of storing the cumulative vector in the cumulative region 611 is performed at least once. For example, Figure 6Corresponding to the following situation: the first accumulated vector is generated by a convolution operation between the first input plane corresponding to the first input channel (e.g., input plane 520 corresponding to input element vectors 5011, 5012, and 5013) and the first weight plane (e.g., weight plane 530), and is stored in the accumulation region 611. The second accumulated vector is generated by a convolution operation between the second input plane corresponding to the second input channel (e.g., input plane corresponding to input element vectors 5021, 5022, and 5023) and the second weight plane (e.g., the second weight plane of weight kernel 510). The first and second accumulated vectors are accumulated and stored in the accumulation region 611. If the initial value is stored in the accumulation region 611 (i.e., if the accumulated vector is generated first), the process of loading the accumulated vector from the accumulation region 611 is omitted, and (e.g., at least since the second accumulated vector has not yet been generated) the newly generated accumulated vector is stored in the accumulation region 611 without performing a separate accumulation operation.

[0068] When the cumulative vector is iteratively stored in the cumulative region 611 a number of times corresponding to the number of input channels, the output element vector 11 corresponding to the cumulative region 611 is determined. For example, for Figure 5 The remaining input elements 5012, 5013, 5022, 5023, 5032, and 5033 are iteratively processed in accordance with the above. Figure 5 The input element vectors 5011, 5021, and 5031 are processed using operations. Furthermore, this operation is performed iteratively for the weight kernels corresponding to different output channels, thereby generating output element vectors 12 to 1D, and thus generating output line 620. As described above, during the generation of output line 620, output line 620 is temporarily stored in an output line buffer, and when the generation of output line 620 is complete, output line 620 is stored in the memory space storing the input feature map, thereby overwriting a portion of the input feature map.

[0069] Figures 7 to 9 An example showing the storage state of the input and output lines in the memory. Figures 7 to 9 The example assumes that the computation block consists of a single convolution operation. Therefore, if an output line is generated, it is immediately stored in memory space. If the computation block includes operations other than a single convolution operation, additional operations related to the output line are performed, and the corresponding results are stored in memory space. Memory space specifically corresponds to the workspace of memory space. Although in Figure 7 In the example, the output line Out_1 is stored in memory space 710. However, if the operation block includes other operations, the additional operation results related to the output line Out_1 are stored in memory space 710, which will be further described below.

[0070] Reference Figure 7 The input feature maps, including input lines In_1 to In_h, are stored in memory space 720. Memory space 710 is reserved for output line Out_1. Memory space 710 is a separate space different from memory space 720. Data for input line In_1 is needed until output line Out_1 is generated through cumulative convolution operation. Therefore, memory space 710, different from memory space 720, is allocated separately to hold the data for input line In_1 until output line Out_1 is generated. Convolution operation associated with the first input line group including input lines In_1 to In_3 is performed, thereby generating output line Out_1. If output line Out_1 is generated, it is stored in memory space 710.

[0071] Reference Figure 8 Output line Out_1 is stored in memory space 810, and the input feature map including input lines In_2 to In_h is stored in memory space 830. Memory space 820 is reserved for output line Out_2. A convolution operation associated with the second group of input lines including input lines In_2 to In_4 is performed, thereby generating output line Out_2. If output line Out_2 is generated, it is stored in memory space 820. During the process of generating output line Out_2, input line In_1 previously stored in memory space 820 is replaced (e.g., overwritten) with output line Out_2. This is because input line In_1 is no longer used after output line Out_1 is generated. Therefore, memory space 820 is reused.

[0072] Reference Figure 9 Output line Out_1 is stored in memory space 910, output line Out_2 is stored in memory space 920, and the input feature map including input lines In_3 to In_h is stored in memory space 940. Memory space 930 is reserved for output line Out_3. A convolution operation associated with the third input line group including input lines In_3 to In_5 is performed, thereby generating output line Out_3. If output line Out_3 is generated, it is stored in memory space 930. During the process of generating output line Out_3, input line In_2 previously stored in memory space 930 is replaced with output line Out_3. This is because if output line Out_2 is generated, input line In_2 is no longer used. Therefore, memory space 930 is reused.

[0073] The above process is performed iteratively until all output lines are generated, thus generating an output feature map. During this process, the output lines of the output feature map are overwritten into the memory space of the input feature map that is no longer used for convolution operations, thereby improving the efficiency of the memory space and thus improving the functionality of the computer on which the processing can be performed.

[0074] Figure 10 Examples of additional memory space allocation based on the relationship between the size of the output feature map and the size of the input feature map are shown. The left side shows the case where the size of the output feature map is equal to or less than the size of the input feature map (hereinafter referred to as Case 1), and the right side shows the case where the size of the output feature map is greater than the size of the input feature map (hereinafter referred to as Case 2). Figures 7 to 9 The example corresponds to the case where the size of the output feature map is equal to the size of the input feature map.

[0075] As described above, Case 1 requires data for input line In_1 until output line Out_1 is completed through cumulative convolution. Therefore, memory space 1010 is additionally allocated to hold the data for input line In_1 until output line Out_1 is generated. Case 2 requires additional memory space 1020, which is larger than memory space 1010.

[0076] In case 1, the amount of additional memory space required is equal to the size of the output line buffer. The size of the output line buffer corresponds to the value obtained by dividing the size of the output feature map by h. Here, h represents the number of input lines and the number of output lines. This can be represented by Equation 1.

[0077] [Equation 1]

[0078]

[0079] In Equation 1, memory a This indicates the amount of additional memory space that needs to be allocated. (map) out This indicates the size of the output feature map. in This indicates the size of the input feature map. h represents the number of input lines and the number of output lines.

[0080] In case 2, the amount of additional memory space required is equal to the value obtained by adding the size of the output line buffer to the difference between the size of the input feature map and the size of the output feature map. This can be represented by Equation 2.

[0081] [Equation 2]

[0082] memory a =map out / h

[0083] In Equation 2, memory a This indicates the amount of additional memory space that needs to be allocated. (map) out This indicates the size of the output feature map. h represents the number of input lines and the number of output lines.

[0084] The processing device performs memory setup and memory allocation based on the memory information of the input feature map and the memory information of the output feature map. For example, the processing device compares the size of the input feature map with the size of the output feature map and determines the size of additional memory (e.g., the size of the additional memory space required) based on one of Equations 1 and 2.

[0085] The processing device determines the memory address of the output feature map based on the size of the additional memory and the memory information of the output feature map. In this example, the memory address of the output feature map is determined such that at least a portion of the output feature map is overwritten by at least a portion of the input feature map. The processing device then determines the memory index for each output line. For example, the processing device performs memory indexing for each output line given the memory address of the output feature map and the number of output lines.

[0086] As described above, an operation block may include at least one operation other than a single convolution operation. For example, an operation block may include a convolution operation (hereinafter referred to as the first operation) and an additional operation (hereinafter referred to as the second operation). The second operation may include an additional convolution operation (e.g., a different convolution operation from the convolution operation in the first operation), skip connections, and / or pooling operations. In this example, the processing device allocates an additional buffer to perform the second operation additionally for the result of the first operation.

[0087] For example, the result of performing the first operation is defined as the first output line, and the result of performing the second operation is defined as the first output. In this example, the processing device operates as follows: The processing device allocates an additional buffer for the first output line. The additional buffer corresponds to the size of the first output line. The processing device performs the first operation and stores the result of the first operation as the first output line in the additional buffer.

[0088] The processing device allocates an output line buffer for the first output. The output line buffer corresponds to the size of the first output. The processing device sequentially or simultaneously allocates additional buffers and output line buffers. The processing device performs a second operation using the first output line and stores the result of the second operation as the first output in the output line buffer. When the first output is complete, the processing device stores the first output from the output line buffer in memory. In this example, a portion of the input feature map is replaced with the first output.

[0089] Through the above processing, the memory space required to perform multiple operations can be minimized. In the following text, reference will be made to... Figure 11 and Figure 12 Describe an example of an operation block that includes multiple operations.

[0090] Figure 11 An example of computational processing related to an operation block that includes skip connections is shown. (See reference...) Figure 11 The operation block 1100 includes a first convolution operation Conv_1, a second convolution operation Conv_2, and a skip connection between the input and output. For example, the operation block 1100 is used for residual CNN.

[0091] According to general operation processing 1110, a first convolution operation is performed on the input feature map, and an intermediate result is generated. The intermediate result corresponds to the output feature map based on the first convolution operation. Then, a second convolution operation is performed on the intermediate result, and skip connections associated with the input feature map are applied to generate the final output. Therefore, according to general operation processing 1110, memory space for the input feature map, memory space for the intermediate result, and memory space for the final output are required.

[0092] According to operation 1120, a first convolution operation is performed on the input feature map, and intermediate results are generated. In this example, a portion of the input lines of the input feature map (which will be used to generate the final output) is used to generate the intermediate results. For example, the final output corresponds to a single line, and the intermediate results correspond to three lines. In this example, five input lines are selectively used to generate a single output line. The three lines corresponding to the intermediate results are stored in an additional buffer. Therefore, memory space for storing all intermediate results, as in general operation 1110, is not required.

[0093] Then, a second convolution operation is performed on the intermediate results, and skip connections associated with the input feature map are applied to generate the final output. The final output is stored in a single output line buffer and then in the memory space used for the input feature map. Therefore, the memory space used to store the entire final output, as in general operation process 1110, is not required. Therefore, according to operation process 1120, the memory space used for the input feature map, additional buffers for the intermediate results (e.g., buffers the size of three lines), and output line buffers for the final output (e.g., buffers the size of a single line) are required. Therefore, memory space efficiency can be improved, thereby improving the functionality of the computer on which operation process 1120 can be implemented.

[0094] Figure 12 This illustrates an example of computation processing associated with a block of operations, including pooling operations. (See also...) Figure 12The operation block 1200 includes convolution (Conv) and pooling operations. For example, the size of the convolution operation is 3×3, and the size of the pooling operation is 2×2.

[0095] According to the general operation processing 1210, a convolution operation is performed on the input feature map, and an intermediate result is generated. The intermediate result corresponds to the output feature map based on the convolution operation. Then, a pooling operation is performed on the intermediate result, and a final output is generated. Therefore, according to the general operation processing 1210, memory space for the input feature map, memory space for the intermediate result, and memory space for the final output are required.

[0096] According to operation 1220, a convolution operation is performed on the input feature map, and intermediate results are generated. In this example, a portion of the input lines of the input feature map (which will be used to generate the final output) is used to generate the intermediate results. For example, the final output corresponds to a single line, and the intermediate results correspond to two lines. In this example, four input lines are selectively used to generate a single output line. The two lines corresponding to the intermediate results are stored in an additional buffer. Therefore, memory space for storing all intermediate results, as in the general operation 1210, is not required.

[0097] Then, pooling operations are performed on the intermediate results, and the final output is generated. The final output is stored in a single output line buffer and then in the memory space used for the input feature map. Therefore, the memory space used to store the entire final output, as in general operation process 1210, is not needed. Therefore, according to operation process 1220, the memory space used for the input feature map, additional buffers for the intermediate results (e.g., buffers the size corresponding to two lines), and output line buffers for the final output (e.g., buffers the size corresponding to a single line) are needed. Thus, the memory space can be used efficiently, thereby improving the functionality of the computer on which operation process 1220 can be implemented.

[0098] Figure 13 This is a flowchart illustrating an example of setting up a memory. (See reference...) Figure 13 In operation 1310, the processing device receives input memory information and output memory information, and performs memory setup and memory allocation. The input corresponds to an input feature map, and the output corresponds to an output feature map or the final computation result. For example, if the computation block includes a single convolution, the output corresponds to the output feature map. If the computation block includes multiple operations, the output corresponds to the final computation result. The memory information includes information related to the memory space used for input and output (e.g., memory address), the size of the input, and the size of the output.

[0099] The processing device performs memory setup and allocation based on the input memory information and the output memory information. For example, the processing device compares the size of the input with the size of the output and determines the size of additional memory based on one of Equations 1 and 2. The processing device determines the memory address of the output based on the size of the additional memory and the memory information of the output. In this example, the memory address of the output is determined such that at least a portion of the output is overwritten onto at least a portion of the input.

[0100] In operation 1320, the processing device performs memory indexing on the output lines. The processing device performs memory indexing on each output line given the memory address of the output and the number of output lines. For example, the processing device divides the memory space of the output based on the memory address of the output by the number of output lines, and performs memory indexing on the output lines based on the result of the division. The processing device then stores the output lines in the memory space based on the result of the memory indexing. In this example, at least a portion of the output lines is overwritten to at least a portion of the inputs.

[0101] Figure 14 This is a flowchart illustrating an example of convolution operation processing. (See also...) Figure 14 In operation 1410, the processing device receives input and obtains the memory indexes of the input lines In_h and the outputs. For example, the processing device obtains a first group of input lines including input lines In_1 to In_3. In operation 1420, the processing device obtains the weight core w_d. For example, the processing device obtains the weight core w_1.

[0102] In operation 1430, the processing device performs a convolution operation. For example, if a first input line group and a weight kernel w_1 are obtained beforehand, the processing device performs a convolution operation between the first input line group and the weight kernel w_1. As a result of performing the convolution operation, a first output line Out_1 is generated. In operation 1440, the processing device stores the output at a target address. Here, the output is the first output line Out_1 or the final result obtained by applying additional operations to the first output line Out_1. For example, the additional operations include additional convolution operations, skip connections, and / or pooling operations. The target address is determined by the memory index of the output.

[0103] In operation 1450, the processing device compares d with D. d represents the index of the current weight kernel, and D represents the total number of weight kernels. If d is not equal to D, the processing device increments d by "1" and executes operation 1420. If d equals D, in operation 1460, the processing device compares h with H. h represents the index of the current input line, and H represents the total number of input lines. If h is not equal to H, the processing device increments h by "1" and executes operation 1410. If h equals H, the convolution operation terminates.

[0104] Figure 15 An example of memory mapping is shown. (See reference...) Figure 15 The memory map includes a text region, an initialized data region, an uninitialized data region, a stack region, a heap region, and an environment variable region. These regions are allocated addresses from low to high. Initialized data regions are represented by data, while uninitialized data regions are represented by symbol-starting blocks (bss).

[0105] In an environment where a neural network is implemented, executable instructions are stored in the text region, and the neural network model is stored in the data region. Data associated with the neural network model includes weight kernels. The stack region, heap region, and BSS region correspond to the workspace. The input feature maps, output feature maps, and intermediate data required for processing neural network-related operations (e.g., convolution operations) all need to be processed in the workspace.

[0106] In environments with mobile hardware dedicated processors or limited memory (such as Internet of Things (IoT) environments), it is difficult to perform all the processing related to neural networks within the workspace. For example, overflows can occur. According to the example, memory space efficiency can be improved because the output of the operation is stored in the memory space used for the input feature map. Therefore, even in environments with limited memory, the probability of overflows can be reduced, thereby improving the functionality of computers capable of memory mapping.

[0107] Figure 16 This is a flowchart illustrating an example of a convolution operation processing method. (See reference...) Figure 16 In operation 1610, the processing device generates a first output line of the output feature map by performing a convolution operation between a first group of input lines of the input feature map and a weight kernel. In operation 1620, the processing device generates a first output based on the first output line, including a block of operations comprising the convolution operation. In operation 1630, the processing device stores the first output in memory by overwriting the first output to a memory space storing the input feature map. Furthermore, Figures 1 to 15 The description can be applied to convolution operation processing methods, so a more detailed description is omitted here.

[0108] Figure 17 An example of a processing device is shown. (See reference) Figure 17The processing device 1700 includes a processor 1710 and a memory 1720. The memory 1720 is connected to the processor 1710 and stores instructions executable by the processor 1710, data to be computed by the processor 1710, or data processed by the processor 1710. The memory 1720 includes non-transitory computer-readable media (e.g., high-speed random access memory) and / or non-volatile computer-readable media (e.g., at least one disk storage device, flash memory device, or other non-volatile solid-state memory device).

[0109] Processor 1710 executes the reference Figures 1 to 16 Instructions for at least one of the described operations. For example, processor 1710 generates a first output line of an output feature map by performing a convolution operation between a first group of input lines of an input feature map and a weight kernel, generates a first output based on the first output line, including a block of operations comprising the convolution operation, and stores the first output in memory by overwriting the first output to a memory space storing the input feature map. Furthermore, Figures 1 to 16 The description applies to processing device 1700, therefore a more detailed description is omitted here.

[0110] In this regard Figures 1 to 17The described processing device 100, processing device 1700, and other devices, units, modules, apparatuses, and other components are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0111] Perform the operations described in this application Figures 1 to 17The methods illustrated are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0112] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Programmers skilled in the art can readily write the instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0113] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), programmable random access read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and to provide instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0114] While this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be construed as included in the disclosure.

Claims

1. An operating method of a neural network processing device for efficiently utilizing memory space, the operating method comprising: storing an input feature map in a memory; generating a first output line of an output feature map by performing a convolution operation between a first input line group of the input feature map and a weight kernel; generating a first output including an operation block of the convolution operation based on the first output line; and storing the first output in the memory by overwriting the first output to a memory space in which the input feature map is stored, wherein the input feature map is stored in the memory in a line data structure to which a data ordering is applied in an order of a width direction, a channel direction, and a height direction. The storing of the first output in the memory includes overwriting the first output to a memory space of at least one input feature element in the input feature map that is not used in a subsequent processing of the operation block.

2. The operating method of claim 1, wherein, The at least one input feature element is included in the first input line group and is not included in a subsequent input line group of the input feature map.

3. The operating method of claim 2, wherein, The subsequent processing of the operation block includes a subsequent convolution operation between the subsequent input line group and the weight kernel.

4. The operating method of claim 3, wherein, The operation block further includes any one or any combination of a pooling operation, a skip connection, and another convolution operation.

5. The operating method of claim 1, wherein, 6.The operating method of claim 1, further comprising: obtaining information related to a memory space of the input feature map; and allocating a memory space for the output feature map such that the output feature map is overwritten to at least a portion of the memory space of the input feature map. The generating of the first output line includes generating a first output element vector of the first output line by performing a convolution operation between an input line of the first input line group and a first weight kernel corresponding to a first output channel, and The first output element vector corresponds to the first output channel.

7. The method of operating according to any one of claims 1 to 6, wherein, The generating of the first output line includes generating the first output line by accumulating a weighted input element vector based on a product between an input element vector of the first input line group and a weight element of the weight kernel. A first input element vector among the input element vectors corresponding to a first offset is multiplied by a first weight element among the weight elements corresponding to the first offset.

8. The method of operation according to any one of claims 1 to 6, wherein, In response to the first weight element being a zero weight element corresponding to "0", a multiplication between the first input element vector and the first weight element is omitted.

9. The operating method of claim 8, wherein, In response to the operation block including a first operation corresponding to the convolution operation and a second operation using the first output line, 10. The operating method of claim 9, wherein, The generating of the first output includes:

11. The method of operating according to any one of claims 1 to 6, wherein, allocating an additional buffer for the first output line, the additional buffer corresponding to a size of the first output line; performing the first operation and storing a result of the first operation as the first output line in the additional buffer; allocating an output line buffer for the first output, the output line buffer corresponding to a size of the first output; and performing the second operation using the first output line and storing a result of the second operation as the first output in the output line buffer. The storing of the first output in the memory includes storing the first output of the output line buffer in the memory. ​ 12. The method of operation of claim 11, wherein, ​ 13. The operating method of claim 1, wherein, The line data structure is different from the plane data structure, and is different from the interleaved data structure, data ordering is applied to the plane data structure in the order of the width direction, the height direction and the channel direction, and data ordering is applied to the interleaved data structure in the order of the channel direction, the width direction and the height direction.

14. The operating method of claim 1, wherein, The input feature map includes a plurality of input lines, each of which includes an input feature vector.

15. The operating method of claim 1, wherein, The first output line includes output feature vectors of a plurality of output channels.

16. The operating method of claim 1, wherein, The number of input lines included in the first input line group corresponds to the height of the weight kernel.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the operation method of claim 1.

18. A neural network processing device for efficiently utilizing memory space, the neural network processing device comprising: a processor configured to: store an input feature map in a memory, generate a first output line of an output feature map by performing a convolution operation between a first input line group of the input feature map and a weight kernel, generate a first output including an operation block of the convolution operation based on the first output line, and store the first output in the memory by overwriting the first output to a memory space in which the input feature map is stored, wherein the input feature map is stored in the memory in a line data structure, and data ordering is applied to the line data structure in the order of a width direction, a channel direction and a height direction.

19. The neural network processing device of claim 18, wherein, The processor is further configured to, when storing the first output in the memory, overwrite the first output to at least one memory space in which an input feature element of the input feature map is no longer used to process the operation block.

20. The neural network processing device of claim 18, wherein, The processor is further configured to obtain information related to the memory space of the input feature map, and allocate a memory space for the output feature map such that the output feature map is overwritten to at least a portion of the memory space of the input feature map.

21. The neural network processing device of any one of claims 18-20, wherein, The processor is further configured to generate the first output line by accumulating multiplication results of input element vectors based on the first input line group and weight elements of the weight kernel.

22. The neural network processing device of any one of claims 18-20, wherein, The processor is further configured to generate a first output element vector of the first output line by accumulating multiplication results of input element vectors based on the first input line group and weight elements of a first weight kernel corresponding to the first output channel in the weight kernel, and The first output element vector corresponds to the first output channel.

23. The neural network processing device of any one of claims 18-20, wherein, In response to the operation block including a first operation corresponding to the convolution operation and a second operation using the first output line, The processor is further configured to allocate an additional buffer for the first output line, the additional buffer corresponding to a size of the first output line; perform the first operation and store a result of the first operation as the first output line in the additional buffer; allocate an output line buffer for the first output, the output line buffer corresponding to a size of the first output; and perform the second operation using the first output line and store a result of the second operation as the first output in the output line buffer.

24. The neural network processing device of claim 23, wherein, The processor is further configured to store the first output of the output line buffer in the memory.

25. The neural network processing device of claim 18, wherein, The input feature map includes a plurality of input lines, each of which includes an input feature vector. The input feature map includes a plurality of input lines, each of which includes an input feature vector.

26. The device of claim 18, further comprising a memory storing instructions that, when executed by the processor, configure the processor to perform the steps of generating the first output line of the output feature map, generating the first output of the operation block, and storing the first output in the memory.

27. A method of operating a neural network processing device for efficient utilization of memory space, comprising: storing an input feature map in a memory; generating a first output line of an output feature map by performing a convolution operation between a first set of input lines of the input feature map and a weight kernel; storing the first output line in a first memory space of the memory; generating a second output line of the output feature map by performing a convolution operation between a second set of input lines of the input feature map and the weight kernel; and storing the second output line in a second memory space of the memory by overwriting the second output line to a second memory space of the memory that includes a first input line of the first set of input lines, wherein the input feature map is stored in the memory in a line data structure to which a data ordering is applied in an order of a width direction, a channel direction, and a height direction. The second set of input lines does not include the first input line of the first set of input lines, but includes remaining input lines of the first set of input lines.

28. The method of operating of claim 27, wherein, ​

Images