Neural network device and method of operating the same

Abstract

A neural network apparatus includes a memory configured to store a first feature map and a second feature map, and a neural network processor configured to operate a neural network and including an acquirer configured to acquire input data from the first feature map of the memory, a buffer configured to store the input data, an operator configured to generate output data by performing a convolution operation between the input data and a kernel, a writer configured to write the output data to the second feature map of the memory, and a controller configured to control the acquirer to acquire the input data and the writer to write the output data according to one or more intervals and one or more offsets determined based on an inflation rate of the kernel in a plurality of steps.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0158933, filed on November 24, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to neural network devices and their operation methods. Background Technology

[0003] Neural networks can be used to analyze input data and extract useful information in various types of electronic systems. Specifically, convolutional neural networks (CNNs) exhibit excellent performance in extracting spatial features, including those in images. However, the convolutional operations used to operate CNNs require high input / output bandwidth. Summary of the Invention

[0004] The present invention is provided in a simplified form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a neural network device includes: a memory configured to store a first feature map and a second feature map; and a neural network processor configured to operate a neural network and including: a acquirer configured to acquire input data from the first feature map in the memory; a buffer configured to store the input data; an arithmetic unit configured to generate output data by performing a convolution operation between the input data and a kernel; a writer configured to write the output data to the second feature map in the memory; and a controller configured to: control the acquirer to acquire the input data and control the writer to write the output data based on one or more intervals and one or more offsets determined in multiple steps based on the kernel's expansion rate.

[0006] To control the acquirer, the controller can be configured to: control the acquirer to acquire input data from a first feature map in memory without redundancy, based on a first interval of one or more intervals determined based on the expansion rate and a first offset of one or more offsets.

[0007] In order to acquire input data, the acquirer can be configured to acquire multiple input data points in a first feature map corresponding to positions reached by skipping a first interval along a predetermined direction from the position of the index corresponding to the first offset.

[0008] In order to control the writer, the controller can be configured to control the writer to write output data to a second feature map of the memory without redundancy, based on a second interval of one or more intervals determined based on the expansion rate and a second offset of one or more offsets.

[0009] In order to write output data, the writer can be configured to write output data at a position in the second feature map that is reached by skipping a second interval in a predetermined direction from the position of the index corresponding to the second offset.

[0010] To acquire input data, the acquirer may be configured to acquire input data from a first feature map of the memory based on a first offset of the one or more offsets and a first interval of the one or more intervals, and to write output data, the writer may be configured to write output data to a second feature map of the memory based on a second offset of the one or more offsets and a second interval of the one or more intervals.

[0011] In response to the stride parameter of the convolution operation being a preset first value, the controller can be configured to: set the first interval and the second interval to have the same value as the value corresponding to the dilation rate; and set the first offset and the second offset to have the same value as the index of the corresponding step among the plurality of steps.

[0012] In response to the stride parameter of the convolution operation being greater than or equal to a preset first value, the controller can be configured to: set the first interval and the second interval to be different from each other; and set the first offset and the second offset to be different from each other.

[0013] To set the first offset and the second offset, the controller can be configured to: set the first offset based on a combination of the step size parameter and the index of the corresponding step among the plurality of steps; and set the second offset to the index of the corresponding step.

[0014] To set the first interval and the second interval, the controller can be configured to: set the first interval to a value that is the same as the value of the expansion rate; and set the second interval to a value obtained by dividing the expansion rate by the greatest common divisor (GCD) of the expansion rate and the step size parameter.

[0015] The controller can be configured to determine the total number of the plurality of steps based on whether a first offset satisfies a first condition and whether a second offset satisfies a second condition. The first condition is that the first offset has the same value as obtained by subtracting the step size parameter from the least common multiple (LCM) of the expansion ratio and the step size parameter. The second condition is that the second offset has the same value as obtained by subtracting "1" from the value obtained by dividing the expansion ratio by the GCD of the expansion ratio and the step size parameter.

[0016] In order to generate output data, in response to the first feature map including multiple channels, the arithmetic unit can be configured to perform dilated convolution operations between the multiple channels and the kernel.

[0017] The spacing between multiple adjacent output data in the second feature map can be determined based on one or two of the following parameters: dilation rate and stride parameter of convolution operation.

[0018] The neural network device may include any one or any combination of two or more of the following: head-up display (HUD) device, three-dimensional (3D) digital information display, navigation system, 3D mobile device, smartphone, smart TV and smart vehicle.

[0019] The memory can be configured to store instructions that, when executed by the neural network processor, configure the neural network processor to operate the neural network.

[0020] In another general aspect, a method of operating a neural network device includes: acquiring input data from a first feature map of memory according to a first interval determined based on a kernel dilation rate; storing the input data in a buffer; generating output data by performing a convolution operation between the input data and a kernel; and writing the output data into a second feature map of memory according to a second interval determined based on a kernel dilation rate.

[0021] The process of acquiring input data may include: acquiring input data from a first feature map without redundancy, based on a first interval and a first offset determined based on the dilation rate.

[0022] The process of acquiring input data may include: acquiring multiple input data points in the first feature map corresponding to the positions reached by skipping a first interval along a predetermined direction from the position of the index corresponding to the first offset.

[0023] The process of writing output data may include: writing the output data into a second feature map in memory without redundancy, based on a second interval and a second offset determined based on the expansion rate.

[0024] The process of writing output data may include: writing output data in the second feature map at a position reached by skipping a second interval along a predetermined direction from the position of the index corresponding to the second offset.

[0025] The process of acquiring input data may include: acquiring input data from a first feature map according to a first interval and a first offset determined based on the expansion rate, and the process of writing output data may include: writing output data into a second feature map in memory according to a second interval and a second offset determined based on the expansion rate.

[0026] The method can be repeatedly performed in multiple steps and may include: setting a first value for the stride parameter in response to the convolution operation, setting a first interval and a second interval to have the same value as the value corresponding to the dilation rate, and setting a first offset and a second offset to have the same value as the index of the corresponding step among the multiple steps.

[0027] The method may include: in response to a stride parameter of the convolution operation being greater than or equal to a preset first value, setting a first interval and a second interval to be different from each other, and setting a first offset and a second offset to be different from each other.

[0028] The method may include: setting a first offset based on a combination of a step size parameter and the index of a corresponding step among a plurality of steps; and setting a second offset to the index of the corresponding step.

[0029] The method may include: setting a first interval to have the same value as the expansion rate; and setting a second interval to a value obtained by dividing the expansion rate by the greatest common divisor (GCD) of the expansion rate and the step size parameter.

[0030] The method can be repeatedly performed in multiple steps, and the number of the multiple steps can be determined based on whether a first offset satisfies a first condition and whether a second offset satisfies a second condition. The first condition is that the first offset has the same value as the value obtained by subtracting the step size parameter from the least common multiple (LCM) of the expansion ratio and the step size parameter. The second condition is that the second offset has the same value as the value obtained by subtracting "1" from the value obtained by dividing the expansion ratio by the GCD of the expansion ratio and the step size parameter.

[0031] The processing of generating output data may include: in response to the first feature map including multiple channels, performing dilated convolution operations between the multiple channels and the kernel.

[0032] The spacing between multiple adjacent output data in the second feature map can be determined based on one or two of the following parameters: dilation rate and stride parameter of convolution operation.

[0033] In another general aspect, one or more embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform any, any combination, or all of the operations and methods described herein.

[0034] In another general aspect, a neural network device includes: one or more processors configured to: determine input data to be acquired in a segment of a first feature map stored in memory based on the kernel expansion rate; acquire the determined input data from memory and store the input data in a buffer; generate output data by performing a convolution operation between the input data and the kernel; and write the output data into a second feature map stored in memory.

[0035] In order to determine the input data to be acquired, the one or more processors may be configured to determine the input data as segments of a first feature map separated by dilation rate.

[0036] In order to perform convolution operations, the one or more processors may be configured to: perform a first convolution operation between a first portion of the input data and a kernel, and perform a second convolution operation between a second portion of the input data and a kernel.

[0037] The second part may include: one or more segments of the first feature map included in the first part; and one or more segments of the first feature map not included in the first part.

[0038] In response to the writing of output data, the one or more processors may be configured to determine subsequent input data to be acquired within a segment of a first feature map stored in memory, based on the expansion rate and step size parameters.

[0039] Subsequent input data may include segments of the first feature map that are different from the segments included in the input data.

[0040] In another general aspect, a method of operating a neural network device includes: determining segments of a first feature map separated by the dilation rate of a kernel; generating a second feature map by performing a convolution operation between the segments and a kernel, wherein the kernel is not dilated by the dilation rate, and the second feature map corresponds to a feature map generated by performing a convolution operation between the first feature map and a kernel dilated by the dilation rate.

[0041] The convolution operation between the segment and the kernel may include a first convolution operation and a second convolution operation, and a portion of the segment used in the first convolution operation may be reused in the second convolution operation.

[0042] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0043] Figure 1 An example of the structure of a neural network is shown.

[0044] Figure 2 Examples of nuclear expansion rate, offset, and spacing are shown.

[0045] Figure 3 An example of a method for performing a one-dimensional (1D) dilated convolution operation is shown.

[0046] Figure 4 An example configuration of a neural network device is shown.

[0047] Figure 5 An example of the operation processing of a neural network device is shown.

[0048] Figure 6 and Figure 7 An example of how a neural network device operates is shown.

[0049] Figure 8 An example of the operation processing of a neural network device is shown.

[0050] Figure 9 An example configuration of a neural network device is shown.

[0051] Figure 10 An example of a method for performing two-dimensional (2D) convolution operations in a neural network device is shown.

[0052] Figure 11 An example of how a neural network device operates is shown.

[0053] Figure 12 An example configuration of a neural network device is shown.

[0054] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0055] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known upon understanding this disclosure may be omitted.

[0056] Although the terms “first” or “second” are used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. For example, without departing from the teaching of the examples described herein, a “first” component, “first” assembly, “first” region, “first” layer, or “first” part may also be referred to as a “second” component, “second” assembly, “second” region, “second” layer, or “second” part.

[0057] Throughout the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, it may be directly "on" said other element, directly "connected to," or directly "bonded to" said other element, or there may be one or more other elements in between. In contrast, when an element is described as being "directly on" another element, directly "connected to," or "bonded to" another element, there may be no other elements in between. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "closely adjacent to" may be interpreted as described above.

[0058] The terminology used herein is for the purpose of describing particular examples only and should not be limiting of the examples. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any one and any combination of any two or more of the associated listed items. As used herein, the terms "comprising," "including," and "having" indicate the presence of the stated features, integrals, steps, operations, elements, components, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof. The use of the term "may" herein with respect to examples or embodiments (e.g., what an example or embodiment may include or implement) indicates the presence of at least one example or embodiment that includes or implements such a feature, while all examples are not limited thereto.

[0059] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in general dictionaries) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalized sense.

[0060] In the following description, examples will be described in detail with reference to the accompanying drawings. When describing examples with reference to the accompanying drawings, the same reference numerals denote the same constituent elements, and repeated descriptions related to them will be omitted.

[0061] Figure 1 An example illustrating the structure of a neural network. (See reference...) Figure 1 The structure of a neural network 100, including convolutional layers and subsampling layers, is shown.

[0062] The neural network 100 may be, for example, a deep neural network (DNN) comprising multiple layers. The DNN may include, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, and / or a restricted Boltzmann machine, but is not limited to these.

[0063] Apart from Figure 1 In addition to the convolutional layers and subsampling layers shown, the neural network 100 may also include fully connected layers. Subsampling layers may also be referred to as "pooling layers".

[0064] The neural network 100 may include multiple layers that receive input data, extract input feature maps, and generate output data from the input feature maps. The neural network 100 can generate an output feature map corresponding to the result obtained by performing a convolution operation between a kernel (hereinafter referred to as a "filter") and the input data. Another convolution operation between the kernel and the generated output feature map, which serves as the input feature map, can be performed, and a new feature map can be output. As a result of repeatedly performing the above convolution operations, a result of the neural network 100 recognizing features of the input image can finally be output. Hereinafter, for ease of description, the input feature map may be referred to as a "first feature map," and the output feature map may be referred to as a "second feature map."

[0065] For example, if an input image of size 24×24 pixels is applied... Figure 1 The input layer 110 of the neural network 100 can output a 20×20 feature map with four channels through convolution operations between the input image and the kernel. Furthermore, subsampling can be used to output a 10×10 feature map with only a subset of pixel values ​​from the 20×20 feature map. Subsampling can include, for example, max pooling and average pooling. Next, the size of the 10×10 feature map can be reduced by repeated convolution operations and repeated subsampling operations between the feature map and the kernel, thus ultimately outputting global features.

[0066] In neural network 100, convolution and subsampling (or pooling) operations can be repeatedly performed in multiple layers, so that robust features representing the entire image can be filtered from the input image applied to input layer 110 and output. Furthermore, the output features, which may correspond to global features, can be input into fully connected layer 160 to be finally extracted as the recognition result of the input image.

[0067] The spatial size of an input neuron that influences a neuron in the output layer (e.g., a fully connected layer 160) can be referred to as the "receptive field" 115. For example... Figure 1 As shown in the lower part, in convolution, the receptive field 115 may correspond to, for example, the middle part of the input layer 110 with an input of 32×32×3.

[0068] In the example below, a dilated convolution (also known as a dilated convolution) neural network can be used. Dilated convolution is a scheme to increase the receptive field by adding zero padding to the filter. The receptive field can be understood as the field of view that the filter considers simultaneously. When the size of the receptive field increases, the filter can more easily extract overall features from the image. However, in a typical CNN, when the size of the receptive field increases, the size of the filter also increases, which leads to an increase in computational cost and overfitting. On the other hand, in a dilated convolution neural network, the size of the receptive field can be increased while maintaining the size of the filter.

[0069] Convolutional operations can utilize high input / output bandwidth, and in typical CNNs, data reuse schemes between adjacent convolutional operations can be used to reduce input / output bandwidth. However, when using dilated convolutional neural networks, typical neural network devices may struggle to reuse data between adjacent convolutional operations. The following examples provide one or more embodiments for implementing data reuse in dilated convolutional neural networks.

[0070] Figure 2 Examples of nuclear dilation rate, offset, and spacing are shown. For example, Figure 2 The structure of each of the following is shown for a kernel of size 3×3: dilated convolution 210 with dilation rate "1", dilated convolution 220 with dilation rate "2", and dilated convolution 230 with dilation rate "3".

[0071] In dilated convolutions 210, 220, and 230, the positions of the input nodes used in a 3×3 kernel can be marked with black boxes. The receptive field of a dilated convolution can vary depending on the dilation rate of the kernel. In one example, when the dilation rate is "1", a 1-dilated 3×3 convolution can be represented as shown in dilated convolution 210. When the dilation rate is "1", the receptive field can have a size of "3×3 = 9", that is, the size of the receptive field can be equal to the size of the kernel.

[0072] In another example, when the dilation rate is "2", a 2-dilated 3×3 convolution can be represented as shown in dilated convolution 220. When the dilation rate is "2", the receptive field can be 5×5 = 25 in size. In another example, when the dilation rate is "3", a 3-dilated 3×3 convolution can be represented as shown in dilated convolution 230. When the dilation rate is "3", the receptive field can be 7×7 = 49 in size. In DNNs, dilated convolutions can be used to classify images as high-resolution images.

[0073] To extract features in a neural network, filters (e.g., kernels) can perform convolution operations while traversing the pixels of the input data or input image at regular intervals (e.g., stride), and feature maps or activation maps can be generated based on the results of the convolution operations. The feature map can be information from the original image compressed by the convolution operation and can be represented, for example, as a matrix. Furthermore, the activation map can correspond to the result obtained by applying an activation function to the feature map. In one example, the activation map can correspond to the final output of a convolutional layer performing convolution operations in a CNN.

[0074] Furthermore, filters may include, for example, common parameters or weight parameters used to find features of the image. A filter may correspond to the kernel described above. In the following text, the terms "filter" and "kernel" are used interchangeably.

[0075] As indicated by reference numeral 250, offset 251 can be understood as a value indicating the starting point of an operation on data stored in memory. For example, to perform a read or write operation on a feature map or data stored in memory, the value indicating the starting point of the operation to read or write data stored in memory may correspond to offset 251. For example, when the offset is "0", a read operation can be performed from the first point of the input data (e.g., the first data in the input data), and when the offset is "1", a read operation can be performed from the second point of the input data (e.g., the second data in the input data).

[0076] When a filter is applied to an input image, the interval at which the filter traverses the pixels of the input image or input data can be called the "stride". For example, when the stride is "2", the filter can perform a convolution operation while traversing the pixels or data of the input image at intervals of two spaces. In this example, the stride parameter or interval can be represented as "2".

[0077] The shape of the final output data from the CNN can be changed based on parameters such as kernel size, kernel dilation rate, and / or stride. Within convolutional layers, the feature map size can be smaller than the input data size due to the effects of filters and stride.

[0078] In one example, when the input image is a color image, each pixel of the feature map may have a value corresponding to each of the three colors (i.e., red (R), green (G), and blue (B)) to represent natural colors. In this example, the first feature map may include three channels. In another example, when the input image is a black and white image, the feature map may represent black and white contrast (instead of representing natural colors using the three colors red (R), green (G), and blue (B),) and the first feature map may include one channel.

[0079] Figure 3 An example of a method for performing a one-dimensional (1D) dilated convolution operation is shown. (See also...) Figure 3 This illustrates the process of performing a dilated convolution operation between input data 301 and kernel 303 of size "3" with a dilation rate of "2".

[0080] For example, there may be eight input data 301 (e.g., "0, 1, 2, 3, 4, 5, 6, 7"), and the stride parameter (e.g., the interval) of the convolution operation may be "1".

[0081] In this example, when the size of kernel 303 is "3" and the inflation rate is "2", as indicated by reference numeral 310, the neural network device can perform a convolution operation between kernel 303 and five of the eight input data 301 that correspond to the size "5" of the inflated kernel 303: "0, 1, 2, 3, 4". As indicated by reference numeral 310, the output data corresponding to the result obtained by performing the convolution operation between kernel 303 and the five input data "0, 1, 2, 3, 4" can be written into the first data of the output feature map 305.

[0082] Subsequently, when the step size parameter (e.g., the movement interval) is "1", kernel 303 can move "1" to the right relative to input data 301, as indicated by reference numeral 320. The neural network device can perform a convolution operation between kernel 303 and five input data lines "1, 2, 3, 4, 5" (corresponding to the position to which kernel 303 has moved). As indicated by reference numeral 320, the result obtained by performing the convolution operation between kernel 303 and the five input data lines "1, 2, 3, 4, 5" can be written into the second data of output feature map 305.

[0083] As indicated by reference numerals 330 and 340, the neural network device can sequentially perform convolution operations between kernel 303 and input data "2, 3, 4, 5, 6" and between kernel 303 and input data "3, 4, 5, 6, 7" while moving kernel 303 to the right by "1". The results obtained by performing the convolution operations can be sequentially written into the third and fourth data of the output feature map 305.

[0084] In a typical convolution operation, the input window used to compute the output data can overlap with the window used to compute adjacent output values. In contrast, the input data used to compute the output data of a dilated convolution operation does not overlap with the input data used to compute adjacent output data, but rather overlaps with the window used to compute output data at positions spaced apart by the dilation rate. In one example, the input data used to compute the first data of output feature map 305 (i.e., "0", "2", and "4") can overlap with the input data used to compute the third data of output feature map 305 (i.e., "2", "4", and "6"), but not with the input data used to compute the second data of output feature map 305 (i.e., "1", "3", and "5").

[0085] In one example, when input windows overlap, data pre-fetched to reduce memory traffic can be buffered in a buffer for reuse. When a buffer is reused as described above, the corresponding buffer can be referred to as a "reuse buffer". In this example, a neural network device of one or more embodiments can perform reuse by buffering while performing computations on data that skips dilated convolutions at the dilation rate. Furthermore, the skipped data can be computed in the same manner as the previous data by adjusting the input offset in the next step.

[0086] Figure 4 An example configuration of a neural network device is shown. (See reference...) Figure 4The neural network device 400 may include one or more memories 410 and a neural network processor 430 (e.g., a convolution engine). The neural network processor 430 may be, for example, a convolution engine configured to perform dilated convolution operations. In the following text, the terms "neural network processor" and "convolution engine" are used interchangeably.

[0087] One or more memories 410 may store a first feature map corresponding to an input storage device and a second feature map corresponding to an output storage device. In one non-limiting example, one or more memories 410 include a memory 410 for both the input storage device and the output storage device. In another non-limiting example, one or more memories 410 may be a memory 410 for both the input storage device and the output storage device.

[0088] The neural network processor 430 may include a acquirer 431, a buffer 433, a processing element 435, a writer 437, and a controller 439 (e.g., one or more processors or neural processors of the neural network processor 430).

[0089] Acquirer 431 can acquire input data from a first feature map in memory 410. Acquirer 431 can send a request for input data to memory 410 and can acquire input data according to the offset and interval set by controller 439. The interval between multiple adjacent input data acquired by acquirer 431 from the input feature map can be determined based on the dilation rate.

[0090] Acquirer 431 can acquire multiple input data points in the first feature map corresponding to "positions reached by skipping a first interval along a predetermined direction from the position corresponding to the index of the first offset". For example, acquirer 431 can acquire multiple input data points in the first feature map corresponding to "positions reached by skipping a first interval along a horizontal direction from the position corresponding to the index of the first offset", and can also acquire multiple input data points in the first feature map corresponding to "positions reached by skipping an interval corresponding to the size of the kernel along a vertical direction from the position corresponding to the index of the first offset". In the following text, "first offset" can be understood as the value indicating the starting point for acquirer 431 to perform a read operation on the input data in the first feature map stored in memory.

[0091] Buffer 433 may store (e.g., buffer) input data acquired by acquirer 431 from the first feature map. Buffer 433 may store the input data acquired by acquirer 431 and may send the stored input data to processing element 435 according to, for example, kernel size and / or step size parameters.

[0092] Processing element 435 generates output data by performing a convolution operation between input data and a kernel. Processing element 435 may correspond to a component configured to perform a convolution operation and may also be referred to as an "operator." In the following text, the terms "processing element" and "operator" are used interchangeably. Processing element 435 can accumulate the product of input data and the kernel and can send output data corresponding to the final result value to writer 437 upon activation.

[0093] For example, when the first feature map includes multiple channels, the processing element 435 can perform a convolution operation between each of the multiple channels and the kernel. For example, when the first feature map includes three channels (e.g., R channel, G channel, and B channel), the processing element 435 can perform a dilated convolution operation between the kernel and each of the three channels (e.g., each of the R channel, G channel, and B channel).

[0094] The writer 437 can write the output data generated by the processing element 435 into the second feature map in the memory 410. The writer 437 can write the output data generated by the processing element 435 into the second feature map in the memory 410 according to the offset and interval determined by the controller 439. For example, the writer 437 can write multiple output data entries in the second feature map at positions reached by skipping a second interval in a predetermined direction from the position corresponding to the second offset index. For example, the writer 437 can write multiple output data entries in the corresponding row of the second feature map at positions reached by skipping a second interval in a horizontal direction from the position corresponding to the second offset index. In this example, the interval between multiple output data entries arranged adjacent to each other in the output feature map can be determined based on one or both of the parameters, such as the dilation rate and the stride of the convolution operation.

[0095] Based on the intervals determined by the core expansion rate, controller 439 can control acquirer 431 to acquire input data from memory 410 without redundancy, and control writer 437 to write output data to memory 410 without redundancy. Controller 439 can determine the offset and interval based on the core expansion rate.

[0096] Based on a first offset and a first interval based on the expansion rate, the controller 439 can control the acquirer 431 to acquire input data from the first feature map in the memory 410 without redundancy.

[0097] Furthermore, based on a second offset and a second interval based on the expansion rate, the controller 439 can control the writer 437 to write the output data into the second feature map in the memory 410 without redundancy. Hereinafter, the "second offset" can be understood as the value indicating the starting point at which the writer 437 performs the operation of writing or storing the output data in the second feature map in the memory 410.

[0098] The controller 439 can control the acquirer 431, buffer 433, processing element 435 and writer 437 to repeatedly perform the above operations in multiple steps based on offset and interval, wherein in each step, the convolution operation between the corresponding input data and the kernel is completed.

[0099] In one example, when the stride parameter of the convolution operation is a preset first value (e.g., "1"), the first and second intervals can be set to have the same values ​​as those corresponding to the dilation rate. In this example, the value corresponding to the dilation rate can be the same as the dilation rate value. The first and second offsets can be set to have the same values ​​as the indices of the corresponding steps in a plurality of steps. The following will refer to... Figure 6 A non-restrictive example describing an operation that involves multiple steps.

[0100] In another example, when the stride parameter of the convolution operation is greater than a first value, the first offset and the second offset can be set to be different from each other, and the first interval and the second interval can also be set to be different from each other. The first offset can be set based on a combination of the stride parameter and the value of the index of the corresponding step among multiple steps, and the second offset can be set to the value of the index of the corresponding step. The first interval can be determined to have the same value as the dilation rate, and the second interval can be determined based on the value obtained by dividing the dilation rate by the greatest common divisor (GCD) of the dilation rate and the stride parameter.

[0101] When the stride parameter of the convolution operation is greater than a first value, the number of iterations of the multiple steps executed by the neural network processor 430 (e.g., processing element 435) can be determined based on whether the first offset satisfies a first condition and whether the second offset satisfies a second condition. The first condition is that the first offset has the same value as "the value obtained by subtracting the stride parameter from the least common multiple (LCM) of the dilation rate and the stride parameter". The second condition is that the second offset has the same value as "the value obtained by subtracting "1" from the value obtained by dividing the dilation rate by the GCD of the dilation rate and the stride parameter".

[0102] According to the example, controller 439 may be included in each of acquirer 431 and writer 437, respectively. For example, controller 439 may include a controller included in acquirer 431 and a controller included in writer 437.

[0103] The neural network device 400 may be or may include any one or any combination of two or more of the following: a head-up display (HUD) device, a three-dimensional (3D) digital information display (DID), a navigation system, a 3D mobile device, a smartphone, a smart TV, and a smart vehicle.

[0104] Figure 5An example of the operation processing of a neural network device is shown. (See reference...) Figure 5 The diagram illustrates the process by which the controller of the neural network device performs dilated convolution operations through operations 510 to 550. The index i of multiple steps can be initialized to "0".

[0105] In operation 510, the controller may send a start signal to each module (e.g., acquirer, buffer, processing element (or arithmetic unit), writer, etc.). In this example, the offset value may be set to the index i of multiple steps, and the interval may be set to the expansion rate of the core. In the accompanying drawings, the expansion rate of the core may be represented as "rate".

[0106] In operation 520, the controller may wait until it receives a signal from each module indicating that the convolution operation is complete according to the above intervals and offsets.

[0107] In operation 530, when a convolution operation completion signal is received from each module, the controller may repeat the above process until the value of index i becomes equal to the value obtained by subtracting "1" from the kernel's dilation rate. In operation 540, the controller may determine whether the value of index i is equal to the value obtained by subtracting "1" from the kernel's dilation rate. When it is determined in operation 540 that the value of index i is equal to the value obtained by subtracting "1" from the kernel's dilation rate, the controller may complete the convolution operation. When it is determined that the value of index i is not equal to (e.g., determined to be different from or less than) the value obtained by subtracting "1" from the kernel's dilation rate, the controller may increment the value of index i by "1" in operation 550, and may perform the dilated convolution operation by repeating the process from operation 510 to operation 540.

[0108] Figure 6 An example illustrating a multi-step operation method for a neural network device is shown. (Refer to...) Figure 6 The diagram illustrates the process by which the neural network device 600 performs a 1D convolution operation through steps 0 and 1 when the stride parameter is "1". In this example, the kernel may have a size of "3" and an expansion rate of "2", and the stride parameter may be "1".

[0109] In one example, when the step size parameter is a first value (e.g., "1"), the controller 639 can set the upper limits of the first and second offsets to values ​​corresponding to the inflation rate (e.g., inflation rate "2" - 1 = 1). In this example, the offset value can be set to the index of the corresponding step. Furthermore, the controller 639 can set the first and second intervals to have the same value as the inflation rate "2". The number of steps in the neural network device 600 can be the same as the inflation rate "2", therefore, steps 0 and 1 can be executed twice.

[0110] The operation of the neural network device 600 in step 0 will now be described. The acquirer 631 can acquire the input data "0, 2, 4, 6" corresponding to the position reached by skipping the first interval (=2) from the first data "0" according to the first offset (=0) and the first interval (=2), where the first data "0" is the position of the index corresponding to the first offset (=0) among the eight data (such as "0, 1, 2, 3, 4, 5, 6, 7") in the corresponding row of the first feature map 613 stored in the memory 610.

[0111] The acquirer 631 can transfer the input data "0, 2, 4, 6" acquired from the memory 610 to the buffer 633. The buffer 633 can store the input data "0, 2, 4, 6".

[0112] The multiplier and accumulator (MAC) 635 performs a convolution operation between a kernel and the input data "0, 2, 4, 6" stored in a buffer 633. For example, the MAC 635 can generate output data by performing a convolution operation between the input data "0, 2, 4, 6" and a kernel of size "3". As described above, according to the example, a technique is provided for achieving data reuse even in dilated convolutional neural networks by acquiring input data based on dilated convolution operations.

[0113] The writer 637 can write output data to a third data "2" corresponding to the position reached by skipping the second interval (=2) from the first data "0", where the first data "0" is the position of the index in the second feature map 616 corresponding to the second offset (=0).

[0114] For example, writer 637 can write the result of the convolution operation between input data "0, 2, 4" and the kernel into the first data of the second feature map 616, and can write the result of the convolution operation between input data "2, 4, 6" and the kernel into the third data of the second feature map 616. In a non-limiting example, the first data and the third data can respectively correspond to the data generated by... Figure 3 Reference numerals 310 and 330 indicate the first and third data of the output feature diagram 305. In this example, the input data "2, 4" can be reused.

[0115] Next, the controller 639 can increment the step index i and execute the operation of step 1.

[0116] In step 1, the acquirer 631 can acquire the input data "1, 3, 5, 7" corresponding to the position reached by skipping the first interval (=2) from the second data "1" according to the first offset (=1) and the first interval (=2), where the second data "1" is the position of the index corresponding to the first offset (=1) among the eight data (such as "0, 1, 2, 3, 4, 5, 6, 7") in the corresponding row of the first feature map 613 stored in the memory 610.

[0117] The acquirer 631 can transfer the input data "1, 3, 5, 7" acquired from the memory 610 to the buffer 633. The buffer 633 can store the input data "1, 3, 5, 7".

[0118] The MAC 635 can perform a convolution operation between the input data "1, 3, 5, 7" stored in the buffer 633 and a kernel. For example, the MAC 635 can generate output data by performing a convolution operation between the input data "1, 3, 5, 7" and a kernel of size "3".

[0119] The writer 637 can write output data to a fourth data "3" corresponding to the position reached by skipping the second interval (=2) from the second data "1", where the second data "1" is the position of the index in the second feature map 616 corresponding to the second offset (=1).

[0120] For example, writer 637 can write the result of the convolution operation between input data "1, 3, 5" and the kernel into the second data of the second feature map 616, and can write the result of the convolution operation between input data "3, 5, 7" and the kernel into the fourth data of the second feature map 616. In a non-limiting example, the second data and the fourth data can respectively correspond to the data generated by... Figure 3 Reference marks 320 and 340 indicate the second and fourth data of the output feature diagram 305. In this example, the input data "3, 5" can be reused.

[0121] When the operations of steps 0 and 1 are completed, all values ​​of the four data can be stored as shown in the second feature figure 619.

[0122] Figure 7 An example illustrating the operation of a neural network device. (See reference...) Figure 7 When the step size parameter is greater than "1", the operation of the neural network device 700 can be executed.

[0123] When a step size parameter with a value greater than "1" is included, the neural network device 700 can perform convolution operations by adjusting the offset and spacing of the acquirer 731 to be different from the offset and spacing of the writer 737.

[0124] When the stride parameter of the convolution operation is greater than a first value (e.g., "1"), the controller 739 may set the first offset Offset_f of the acquirer 731 and the second offset Offset_w of the writer 737 to be different from each other. For example, the controller 739 may set the first offset based on a combination (e.g., a product) of the multistep index and the stride parameter, and may set the second offset to the multistep index.

[0125] Furthermore, the controller 739 can set the first interval Interval_f of the acquirer 731 and the second interval Interval_w of the writer 737 to be different from each other. The controller 739 can determine the first interval Interval_f to have the same value as the expansion rate r, and can determine the second interval Interval_w to have the value "r / gcd(r,s)" obtained by dividing the expansion rate r by "the GCD of the expansion rate r and the step size parameter s".

[0126] The controller 739 can control the acquirer 731 to acquire input data from the first feature map of the memory 710 without redundancy according to the first offset_f and the first interval_f, so that the input data can be stored in the buffer 733. The controller 739 can control the buffer 733 to send the stored input data to the MAC 735.

[0127] The controller 739 controls the MAC 735 to generate output data by performing a convolution operation between the core and the input data received from the buffer 733, and then transmits the output data to the writer 737.

[0128] The controller 739 can control the writer 737 to write the output data into the second feature map (not shown) in the memory 710 without redundancy according to the second offset_w and the second interval_w. If the convolution operation is repeatedly performed by the neural network device 700, i can be increased by the stride as much as j, and j can be increased by "1".

[0129] Figure 8 An example of the operation processing of a neural network device is shown. (See reference...) Figure 8 This shows that when the step size parameter is greater than "1", the controller of the neural network device performs the dilated convolution operation through operations 810 to 850.

[0130] In operation 810, the controller may send a start signal to each module (e.g., acquirer, buffer, processing element (or arithmetic unit), writer, etc.). In this example, the value of the first offset_f may be set to "i", and the value of the first interval_f may be set to be the same as the expansion rate r of the core. Furthermore, the value of the second offset_w may be set to "j", and the second interval_w may be set to the value "r / gcd(r,s)" obtained by dividing the expansion rate r by the GCD of the expansion rate r and the step size parameter s. Here, i may represent the product of the multi-step index and the step size parameter, and j may represent the multi-step index.

[0131] In operation 820, the controller may wait until it receives a signal from each module indicating that the convolution operation is complete according to the above intervals and offsets.

[0132] When a convolution operation completion signal is received from each module in operation 830, in operation 840, the controller may determine whether the value i set to the first offset satisfies the first condition and whether the value j set to the second offset satisfies the second condition. The first condition is that i has the same value as the value "lcm(r,s)-s" obtained by subtracting the step size parameter s from the LCM of the dilation rate r and the step size parameter s. The second condition is that j has the same value as the value "r / (gcd(r,s))-1" obtained by subtracting "1" from the value obtained by dividing the dilation rate r by the GCD of the dilation rate r and the step size parameter s.

[0133] In one example, when it is determined in operation 840 that the first offset and the second offset satisfy the first condition and the second condition, respectively, the controller can perform the convolution operation. In another example, when it is determined in operation 840 that the first offset and the second offset do not satisfy the first condition and the second condition, respectively, the controller can increase the value of i by "s" and increase the value of j by "1" in operation 850. Subsequently, the controller can perform the convolution operation by repeatedly executing the processing from operation 810 to operation 840.

[0134] Figure 9 An example configuration of a neural network device is shown. (See reference...) Figure 9 The diagram illustrates the configuration of a neural network device 900 in which a controller is included in each of the acquirer 931 and the writer 937.

[0135] In one example, a first controller 932 configured to control the operation of acquirer 931 and a second controller 938 configured to control the operation of writer 937 may be configured. The first controller 932 may control acquirer 931 to acquire input data from a first feature map (not shown) in memory 910 without redundancy according to a first offset_f and a first interval_f, such that the input data can be stored in buffer 933.

[0136] The second controller 938 can control the writer 937 to write the output data of the processing element 935 into the second feature map (not shown) in the memory 910 without redundancy according to the second offset_w and the second interval_w.

[0137] In this example, the tasks assigned to buffer 933 and processing element 935 can be used for the overall dilated convolution operation, rather than for a single step.

[0138] Figure 10 An example is shown of a method for performing two-dimensional (2D) convolution operations in a neural network device. (See also...) Figure 10 This illustrates the process of performing a 2D dilated convolution operation between input data 1010 and a kernel of size 3×3 with a dilation rate of "2".

[0139] In step 0, the neural network device can operate by setting a first offset and a second offset to “0” and a first interval and a second interval to “2” along the row direction in the first feature map 1010 and the second feature maps 1050 and 1090 in the memory.

[0140] In this example, the acquirer can acquire input data from the row in the first feature map 1010 where the computation will be performed, following the column direction with an interval of two spaces corresponding to the first interval, in accordance with the kernel size of "3". For example, the acquirer can acquire input data ['0,0', '2,0', '4,0'], which corresponds to the position reached by skipping the first interval (=2) corresponding to the kernel size (=3) along the column direction from the position (0,0) of the index corresponding to the first offset in the first row where the operation will be performed. In the same way, the acquirer can acquire input data ['0,2', '2,2', '4,2'], ['0,4', '2,4', '4,4'], ['0,6', '2,6', '4,6'] and ['0,8', '2,8', '4,8']. Although an example of prioritizing the acquisition of input data along the column direction has been described above for ease of description, the design can be modified according to the implementation so that input data is acquired primarily along the row direction.

[0141] The input data acquired by the acquirer from the first feature map 1010 can be stored in the buffer 1030, and the writer can write the result of the convolution operation between the kernel and the input data stored in the buffer 1030 as output data into the second feature map 1050. In this example, the writer can sequentially write the output data (0,0), (0,2), (0,4) by skipping the second interval (=2) for the corresponding rows of the second feature map 1050.

[0142] When all rows have been processed in step 0, the neural network device can proceed to the next step, namely step 1. In step 1, the neural network device can set the first offset and the second offset to "1" and can repeat the above processing.

[0143] For example, the acquirer can acquire input data ['0,1', '2,1', '4,1'], ['0,3', '2,3', '4,3'], ['0,5', '2,5', '4,5'], ['0,7', '2,7', '4,7'], and ['0,9', '2,9', '4,9']. The input data acquired by the acquirer from the first feature map 1010 can be stored in the buffer 1070, and the writer can write the result of the convolution operation between the kernel and the input data stored in the buffer 1070 as output data to the second feature map 1090. In this example, the writer can sequentially write the output data (0,1), (0,3), (0,5) by skipping the second interval (=2) for the corresponding rows of the second feature map 1090. The neural network device can also perform a dilated convolution operation on each row in step 1.

[0144] Figure 11 An example illustrating the operation of a neural network device. (See reference...) Figure 11 Through operations 1110 to 1150, the neural network device can acquire data stored in memory, perform convolution operations, and write the operation results as output data into memory.

[0145] In operation 1110, the neural network device may retrieve input data from a first feature map in memory according to a first interval based on the kernel expansion rate. The neural network device may retrieve input data from the first feature map without redundancy according to the first interval based on the expansion rate and a first offset. In operation 1120, the neural network device may store the input data in a buffer.

[0146] In operation 1130, the neural network device can generate output data by performing convolution operations between the input data and the kernel. For example, when the first feature map includes multiple channels, the neural network device can perform convolution operations between multiple channels and the kernel.

[0147] In operation 1140, the neural network device may write output data into a second feature map in memory. The spacing between multiple output data lines arranged adjacent to each other in the second feature map may be determined based on at least one of, for example, the dilation rate and stride parameter of a convolution operation.

[0148] In operation 1150, the neural network device may write output data into a second feature map in memory according to a second interval based on the kernel expansion rate. The neural network device may write the output data into the second feature map in memory without redundancy according to the second interval and a second offset based on the expansion rate. For example, the neural network device may write the output data into the second feature map at a position reached by skipping the second interval from the position corresponding to the second offset.

[0149] Figure 12 An example configuration of a neural network device is shown. (See reference...) Figure 12 The diagram illustrates the structure of a neural network device including a memory 1210, a buffer 1230, and a MAC 1250. In this example, the buffer 1230 may correspond to a reuse buffer.

[0150] Neural network devices can use reused buffers to improve bandwidth. However, memory costs arise when performing dilated convolution operations in high-performance neural network processors. This is because high-performance neural network processors are required to perform dilated convolution operations, and the required memory bandwidth increases with the computational power of the neural network processor.

[0151] In one example, buffer 1230 can be used as a reuse buffer, so that neural network devices of one or more embodiments can perform dilated convolution operations using small-sized buffers and increase bandwidth even without using additional memory.

[0152] Neural network devices may be, or may include, for example, HUD devices, 3D DIDs, navigation systems, 3D mobile devices, smartphones, smart TVs, and / or smart vehicles, but are not limited thereto. 3D mobile devices may be interpreted as including all display devices (such as, for example, display devices for displaying augmented reality (AR), virtual reality (VR), and / or mixed reality (MR), head-mounted displays (HMDs), face-mounted displays (FMDs), etc.).

[0153] Regarding Figures 1 to 12The described neural network devices include: a memory, a neural network processor, a acquirer, a buffer, a processing element, a writer, a controller, a MAC, a first controller, a second controller, a neural network device 400, a memory 410, a neural network processor 430, a acquirer 431, a buffer 433, a processing element 435, a writer 437, and a controller 439; a neural network device 600, a memory 610, a neural network processor 630, a acquirer 631, a buffer 633, a MAC 635, a writer 637, and a controller 639; a neural network device 700, a memory 710, a neural network processor 730, a acquirer 731, a buffer 733, a MAC 735, a writer 737, and a controller 639; a neural network device 900, a memory 910, a neural network processor 930, a acquirer 931, a first controller 932, a buffer 933, a processing element 935, a writer 937, and a second controller 938; a memory 1210, a buffer 1230, and a MAC. 1250 and other devices, apparatuses, units, modules, and components are implemented by or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller.One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0154] Figures 1 to 12 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0155] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0156] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or may be recorded, stored, or fixed on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, enabling one or more processors or computers to execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions or software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0157] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

Claims

1. A neural network device, comprising: The memory is configured to store the first feature map and the second feature map; as well as A neural network processor, configured to operate a neural network, and includes: The acquirer is configured to acquire input data from a first feature map in memory. The buffer is configured to store input data. The arithmetic unit is configured to generate output data by performing convolution operations between the input data and the kernel. The writer is configured to write output data to a second feature map in memory, and The controller is configured to: control the acquirer to acquire input data to be stored in a buffer from a first feature map of the memory and control the writer to write output data generated by the arithmetic unit to a second feature map of the memory, based on one or more intervals and one or more offsets determined in multiple steps based on the expansion rate of the kernel.

2. The neural network device according to claim 1, wherein, In order to control the acquirer, the controller is configured to: control the acquirer to acquire input data from a first feature map in memory without redundancy, based on a first offset of one or more offsets determined based on the expansion rate and a first interval of one or more intervals.

3. The neural network device according to claim 2, wherein, In order to acquire input data, the acquirer is configured to acquire multiple input data points in the first feature map corresponding to positions reached by skipping a first interval along a predetermined direction from the position of the index corresponding to the first offset.

4. The neural network device according to claim 1, wherein, In order to control the writer, the controller is configured to control the writer to write output data to a second feature map of the memory without redundancy, based on a second offset of one or more offsets determined based on the expansion rate and a second interval of one or more intervals.

5. The neural network device according to claim 4, wherein, In order to write output data, the writer is configured to write output data at a position in the second feature map that is reached by skipping a second interval in a predetermined direction from the position of the index corresponding to the second offset.

6. The neural network device according to claim 1, wherein, To acquire input data, the acquirer is configured to: acquire input data from a first feature map of memory based on a first offset of the one or more offsets and a first interval of the one or more intervals, and In order to write output data, the writer is configured to write the output data to a second feature map of the memory according to a second offset of one or more offsets and a second interval of one or more intervals.

7. The neural network device according to claim 6, wherein, In response to the stride parameter of the convolution operation being a preset first value, the controller is configured as follows: Set the first and second intervals to have the same values ​​as those corresponding to the expansion rate; and Set the first offset and the second offset to the same values ​​as the indices of the corresponding steps in the plurality of steps.

8. The neural network device according to claim 6, wherein, In response to the stride parameter of the convolution operation being greater than or equal to a preset first value, the controller is configured as follows: Set the first interval and the second interval to be different from each other; and Set the first offset and the second offset to be different from each other.

9. The neural network device according to claim 8, wherein, To set the first and second offsets, the controller is configured as follows: The first offset is set based on a combination of the step size parameter and the index of the corresponding step among the plurality of steps; and Set the second offset to the value of the index of the corresponding step.

10. The neural network device according to claim 8, wherein, To set the first and second intervals, the controller is configured as follows: Set the first interval to have the same value as the expansion rate; and The second interval is set to a value obtained by dividing the expansion rate by the greatest common divisor of the expansion rate and the step size parameter.

11. The neural network device according to claim 8, wherein, The controller is configured to determine the total number of the plurality of steps based on whether a first offset satisfies a first condition and whether a second offset satisfies a second condition. The first condition is that the first offset has the same value as obtained by subtracting the step size parameter from the least common multiple of the expansion ratio and the step size parameter. The second condition is that the second offset has the same value as obtained by subtracting "1" from the value obtained by dividing the expansion ratio by the greatest common divisor of the expansion ratio and the step size parameter.

12. The neural network device according to claim 1, wherein, In order to generate output data, in response to the first feature map including multiple channels, the arithmetic unit is configured to perform dilated convolution operations between the multiple channels and the kernel.

13. The neural network device according to claim 1, wherein, The spacing between multiple output data lines arranged adjacent to each other in the second feature map is determined based on one or two of the following parameters: dilation rate and stride parameter of convolution operation.

14. The neural network device according to claim 1, wherein, The neural network device includes any one or any combination of two or more of the following: head-up display device, three-dimensional digital information display, navigation system, three-dimensional mobile device, smartphone, smart TV and intelligent transportation vehicle.

15. The neural network device according to claim 1, wherein, The memory is configured to store instructions that, when executed by the neural network processor, configure the neural network processor to operate the neural network.

16. A method of operating a neural network device, the method comprising: Input data is obtained from a first feature map of the memory based on a first interval determined by the kernel-based expansion rate; Store the input data in a buffer; Output data is generated by performing a convolution operation between the input data and the kernel; as well as The output data is written into the second feature map of the memory based on the second interval determined by the kernel expansion rate.

17. The operating method according to claim 16, wherein, The steps for acquiring input data include: acquiring input data from a first feature map without redundancy, based on a first interval and a first offset determined based on the dilation rate.

18. The operating method according to claim 17, wherein, The steps for obtaining input data include: obtaining multiple input data points in the first feature map corresponding to the positions reached by skipping a first interval along a predetermined direction from the position of the index corresponding to the first offset.

19. The operating method according to claim 16, wherein, The step of writing the output data includes: writing the output data into a second feature map in memory without redundancy, based on a second interval and a second offset determined based on the expansion rate.

20. The operating method according to claim 19, wherein, The step of writing output data includes: writing output data in the second feature map at a position reached by skipping a second interval along a predetermined direction from the position of the index corresponding to the second offset.

21. The operating method according to claim 16, wherein, The steps for acquiring input data include: acquiring input data from a first feature map based on a first interval and a first offset determined based on the dilation rate; and The step of writing the output data includes: writing the output data into a second feature map of the memory according to a second interval and a second offset determined based on the expansion rate.

22. The operating method according to claim 21, in, The operation method is repeatedly performed in multiple steps, and The operation method further includes: in response to the stride parameter of the convolution operation being a preset first value, setting the first interval and the second interval to have the same value as the value corresponding to the dilation rate, and setting the first offset and the second offset to have the same value as the index of the corresponding step among the plurality of steps.

23. The operating method according to claim 21, further comprising: In response to the stride parameter of the convolution operation being greater than or equal to a preset first value, the first interval and the second interval are set to be different from each other, and the first offset and the second offset are set to be different from each other.

24. The operating method according to claim 23, further comprising: The first offset is set based on a combination of the step size parameter and the index of the corresponding step among multiple steps; as well as Set the second offset to the value of the index of the corresponding step.

25. The operating method according to claim 23, further comprising: Set the first interval to have the same value as the expansion rate; as well as The second interval is set to a value obtained by dividing the expansion rate by the greatest common divisor of the expansion rate and the step size parameter.

26. The operating method according to claim 23, wherein, The operation method is repeatedly performed in multiple steps, and The number of the plurality of steps is determined based on whether the first offset satisfies the first condition and whether the second offset satisfies the second condition. The first condition is that the first offset has the same value as the value obtained by subtracting the step size parameter from the least common multiple of the expansion ratio and the step size parameter. The second condition is that the second offset has the same value as the value obtained by subtracting "1" from the value obtained by dividing the expansion ratio by the greatest common divisor of the expansion ratio and the step size parameter.

27. The operating method according to claim 16, wherein, The steps for generating output data include: in response to the first feature map including multiple channels, performing a dilated convolution operation between the multiple channels and the kernel.

28. The operating method according to claim 16, wherein, The spacing between multiple output data lines arranged adjacent to each other in the second feature map is determined based on one or two of the following parameters: dilation rate and stride parameter of convolution operation.

29. A non-transitory computer-readable storage medium for storing instructions, which, when executed by a processor, configure the processor to perform the operation method according to claim 16.

30. A neural network device, comprising: One or more processors are configured as follows: Based on the kernel expansion rate, determine one or more intervals and one or more offsets for obtaining input data within a segment of a first feature map stored in memory; Input data is retrieved from memory and stored in a buffer according to one or more intervals and one or more offsets; Output data is generated by performing a convolution operation between the input data and the kernel; as well as The output data is written into the second feature map in memory.

31. The neural network device according to claim 30, wherein, In order to acquire input data, the one or more processors are configured to determine the input data as segments of a first feature map separated by dilation rate.

32. The neural network device according to claim 30, wherein, In order to perform convolution operations, the one or more processors are configured to perform a first convolution operation between a first portion of the input data and a kernel, and a second convolution operation between a second portion of the input data and a kernel.

33. The neural network device according to claim 32, wherein, The second part includes: The first feature map includes one or more segments in the first portion; and One or more segments of the first feature map that are not included in the first part.

34. The neural network device according to claim 30, wherein, In response to the writing of output data, the one or more processors are configured to determine subsequent input data to be acquired within a segment of a first feature map stored in memory, based on the expansion rate and step size parameters.

35. The neural network device according to claim 34, wherein, Subsequent input data includes segments of the first feature map that are different from the segments included in the input data.

36. A method of operating a neural network device, the method comprising: Identify the segments of the first feature map separated by the expansion rate of the kernel; Based on the kernel expansion rate, determine one or more intervals and one or more offsets of the input data in the segment used to obtain the first feature map; Input data is obtained based on the one or more intervals and the one or more offsets; as well as A second feature map is generated by performing a convolution operation between the input data and a kernel, wherein the kernel is not dilated at a dilation rate, and the second feature map corresponds to a feature map generated by performing a convolution operation between the first feature map and a kernel dilated at a dilation rate.

37. The operating method according to claim 36, wherein, The convolution operation between the input data and the kernel includes a first convolution operation and a second convolution operation, and a portion of the segment used in the first convolution operation is reused in the second convolution operation.

Images