Data transmission method of an acquirer and acquirer

Abstract

A data transmission method of a fetcher and the fetcher are disclosed. The fetcher includes a loader, at least one transmitter, a buffer controller, and a reuse buffer. The data transmission method includes loading, by the loader, input data of an input feature map according to a loading order based on the input data stored in the reuse buffer, a shape of a kernel for a convolution operation, and two-dimensional (2D) zero value information of weights of the kernel, storing, by the buffer controller, the loaded input data in the reuse buffer to which an address is cyclically allocated according to the loading order, and selecting, by each of the at least one transmitter, input data corresponding to each output data of the convolution operation among the input data stored in the reuse buffer based on one-dimensional (1D) zero value information of the weights, and outputting the selected input data.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0180967, filed on December 22, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to buffer management devices, and more specifically, to efficient buffer management devices for data reuse in neural accelerators. Background Technology

[0003] Deep learning is a technique used to train neural networks with multiple layers, each containing multiple neurons, based on a large amount of training data. To improve the accuracy of inferences within a neural network, a large amount of training data is required, and this training data can include images, sound, or text information.

[0004] Convolutional Neural Networks (CNNs) can significantly improve the accuracy of image classification and recognition through convolution operations. However, CNN-based models require a large amount of computation, and the resources required increase as the amount of training data grows.

[0005] Various studies are underway to accelerate convolution operations, with hardware acceleration achieved through hardware improvements. For example, neural processing units (NPUs) are processors designed for parallel processing of matrix operations (such as convolution) and exhibit higher processing speeds than general-purpose processors. Summary of the Invention

[0006] This summary is provided to introduce, in a simplified form, the selection of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0007] In one general aspect, a data transmission method for an acquirer, the acquirer including a loader, at least one transmitter, a buffer controller, and a reuse buffer, the data transmission method comprising: the loader loading input data of an input feature map stored in memory according to a loading order, based on input data stored in the reuse buffer, two-dimensional (2D) zero-value information of the shape of the kernel and the weights of the kernel for a convolution operation; the buffer controller storing the loaded input data in the reuse buffer, the addresses of which are cyclically allocated according to the loading order; and the at least one transmitter selecting input data corresponding to each output data of the convolution operation from the input data stored in the reuse buffer based on one-dimensional (1D) zero-value information of the weights, and sending the selected input data to an executor.

[0008] The kernel may have a rectangular shape. 2D zero-value information may include 2D position information indicating the location of one or more weights, each of which has a zero value.

[0009] The kernel can have shapes other than rectangular shapes. 2D zero-value information may include 2D position information indicating the location of one or more weights that do not overlap with the kernel within the smallest rectangle that overlaps with the kernel.

[0010] The kernel may have a rectangular shape. 2D zero-value information may include 2D position information indicating the location of one or more weights that were removed through pruning.

[0011] The steps of loading input data may include: selecting the location of weights with non-zero values ​​among the weights based on the shape of the kernel and 2D zero-value information; selecting input data that does not overlap with input data stored in a reuse buffer from the input data of the input feature map corresponding to the location of the weights with non-zero values; and loading the selected input data.

[0012] The steps of selecting input data may include: selecting the position of a weight with a non-zero value from the weights based on 1D zero-value information; selecting the input data corresponding to the position of the weight with a non-zero value from the input data stored in the reuse buffer; and sending the selected input data to the actuator.

[0013] A non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform a data sharing method.

[0014] In another general aspect, an apparatus includes: a loader; at least one transmitter; a buffer controller; and a reuse buffer, wherein the loader is configured to: load input data of an input feature map stored in memory according to a loading order, based on input data stored in the reuse buffer, 2D zero-value information of the shape of the kernel to be used for convolution operations, and the weights of the kernel; the buffer controller is configured to: store the loaded input data in the reuse buffer whose addresses are cyclically allocated according to the loading order; and each of the at least one transmitter is configured to: select input data corresponding to each output data of the convolution operation from the input data stored in the reuse buffer based on the 1D zero-value information of the weights, and send the selected input data to an executor.

[0015] The kernel may have a rectangular shape. 2D zero-value information may include 2D position information indicating the location of one or more weights, each of which has a zero value.

[0016] The kernel can have shapes other than rectangular shapes. 2D zero-value information may include 2D position information indicating the location of one or more weights that do not overlap with the kernel within the smallest rectangle that overlaps with the kernel.

[0017] The kernel may have a rectangular shape. 2D zero-value information may include 2D position information indicating the location of one or more weights that were removed through pruning.

[0018] The loader can be configured to: select the position of a weight with a non-zero value among the weights based on the shape of the kernel and 2D zero-value information; select input data that does not overlap with the input data stored in the reuse buffer from the input data of the input feature map corresponding to the position of the weight with a non-zero value; and load the selected input data.

[0019] The at least one transmitter may be configured to: select the position of a weight with a non-zero value from among the weights based on 1D zero-value information; select input data corresponding to the position of the weight with a non-zero value from among the input data stored in a reuse buffer; and send the selected input data to the actuator.

[0020] The device may include: a memory configured to store input feature maps; and an actuator configured to perform parallel convolution operations on selected input data output from the at least one transmitter.

[0021] In another general aspect, an apparatus includes one or more processors configured to: load input data of an input feature map stored in memory by loading feature values ​​of an input feature map corresponding to the positions of non-zero values ​​of the weights of the kernel to be used for convolution operations and skipping feature values ​​of the input feature map corresponding to the positions of zero values ​​of the weights of the kernel; store the input data in a reuse buffer; and select a portion of the input data to be output stored in the reuse buffer based on one-dimensional (1D) zero-value information of the kernel weights.

[0022] In the case that the kernel has a non-rectangular shape, the one or more processors may be configured to assign zero values ​​to any weights that do not overlap with the kernel in the smallest rectangle that completely contains the kernel.

[0023] Other features and aspects will become clear from the following detailed description, the accompanying drawings, and the claims. Attached Figure Description

[0024] Figure 1 An example configuration of a convolution operation device is shown.

[0025] Figure 2 An example of the operation of the data transmission method used for convolution operations is shown.

[0026] Figure 3 An example of a data transmission method used for convolution operations is shown.

[0027] Figure 4 This shows an example of input data loaded by a convolution operation device.

[0028] Figure 5 Examples of one-dimensional (1D) zero-value information and two-dimensional (2D) zero-value information used by convolution operation devices are shown.

[0029] Figure 6 An example of the acquirer configuration is shown.

[0030] Figure 7 This shows another example of the configuration of a convolution operation device.

[0031] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0032] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0033] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0034] The following structural or functional descriptions of the examples disclosed in this disclosure are intended for illustrative purposes only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but are intended to cover various modifications, equivalents, and substitutions within the scope of the claims.

[0035] Although the terms “first” or “second” are used to describe various components, the components are not limited by the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the conception of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0036] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with that other component, or there may be an intermediate component.

[0037] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As used herein, the terms "comprising," "including," and "having" indicate the presence of the described features, quantities, operations, elements, components, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, elements, components, and / or combinations thereof.

[0038] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0039] The examples will be described in detail below with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be designated by the same reference numerals, and redundant descriptions will be omitted.

[0040] Figure 1 An example configuration of a convolution operation device is shown.

[0041] Convolution operations can involve a product-sum operation between a kernel and an input feature map (IFM). The kernel performs this product-sum operation while sliding across the input feature map. The interval at which the kernel slides is called the "stride." Each pixel of the output feature map, the result of the convolution operation, can be the result of a convolution operation of the kernel with respect to a predetermined region of the input feature map. Convolution operations constitute a relatively large portion of the operations in a neural network used to process images. The kernel can also be referred to as a "window."

[0042] In one example, the convolution operation device can accelerate convolution operations by reusing input data based on kernel zero values. When the kernel weights have zero values, the loader 110 and transmitter 140 of the acquirer 100 for efficient convolution operations can store the zero-value information of the weights and can perform convolution operations based on the stored zero-value information. The convolution operation device can perform zero skipping when loading input data and sending input data from transmitter 140 to executor 150.

[0043] For this purpose, the convolution operation device may include a memory (not shown), a acquirer 100, and an executor 150. The acquirer 100 may include a loader 110, at least one transmitter 140, a buffer controller (not shown), and a reuse buffer 130. The executor 150 may also be referred to as an "execution unit".

[0044] The memory can store the input feature map 101. The input feature map 101 can be a multidimensional vector of feature values ​​of the input image. In the two-dimensional (2D) case, the input feature map 101 can include feature values ​​of the input image for each pixel.

[0045] The convolution operation device can perform processing such as acquiring input data from memory, analyzing the acquired input data, and performing operations on the analyzed data. The function of acquiring input data can be performed by the loader 110 of the acquirer 100. The acquirer 100 can also be referred to as an "acquisition unit (FU)".

[0046] The loader 110 can perform memory accesses to load input data stored in memory. The loader 110 can load the input data that will actually be used for the convolution operation based on the location of weights with zero values ​​among multiple weights included in the kernel, and can store the input data in the reuse buffer 130. Because zero values ​​are not loaded, the loader 110 reduces the number of memory accesses and saves space in the reuse buffer 130.

[0047] Information about the zero values ​​of the weights stored in the loader 110 may be referred to as "2D zero value information". 2D zero value information may include 2D positional information about the weights with zero values ​​in the kernel. 2D positional information may be represented by height and width.

[0048] Reference Figure 1 The loader 110 can load input data from the input feature map 101 based on 2D zero-value information 171. The kernel may include weights, where a, b, c, and i are zero values, and d, e, f, g, and h are non-zero values. The 2D zero-value information 171 may include the shape of the kernel and information about the positions of a, b, c, and i.

[0049] The loader 110 can skip the feature values ​​of the input feature map 101 corresponding to the positions of the kernel's zero values ​​based on the 2D zero-value information 171. The kernel can perform convolution operations with the feature values ​​of the input feature map 101 while sliding from top to bottom with a stride.

[0050] For example, when the kernel corresponds to 0 through 8, loader 110 can skip 0, 1, and 2 corresponding to a, b, and c, and can load 3 through 7 corresponding to d, e, f, g, and h. The kernel can slide down by a step "1", and loader 110 can skip 3, 4, and 5 corresponding to a, b, and c, and can load 8 through 10 by skipping already loaded 6 and 7 among 6 through 10 corresponding to d, e, f, g, and h. loader 110 can skip 11 corresponding to i. When loading is repeated as described above, loader 110 can skip 0, 1, and 2 corresponding to a, b, and c, and 17 corresponding to i. loader 110 can load 3 through 16 corresponding to d, e, f, g, and h once. The loaded input data can be stored in reuse buffer 130.

[0051] The acquirer 100 can reduce the number of memory accesses by reusing data. The acquirer 100 can share memory bandwidth resources with one or more executors 150, thereby mitigating memory bottlenecks. The reuse buffer 130 can be a space for storing input data read from the loader 110. The input data may include the input feature map 101.

[0052] The buffer controller can calculate the address where input data will be stored, write the input data to the calculated address, and flush the allocated address. For example, the buffer controller can calculate the address to be allocated to the reuse buffer 130 based on the load counter, the load unit, the size of the reuse buffer 130, and the amount of data to be shared. The buffer controller can calculate the address to be flushed based on the transmit counter of each transmitter 140.

[0053] The transmitter 140 can send input data stored in the reuse buffer 130 to the executor 150. The transmitter 140 can retrieve input data corresponding to each output data of the convolution operation from the reuse buffer 130 and send the input data to the executor 150. The transmitter 140 can select the required input data for the reuse buffer 130 based on the position of weights with zero values ​​among multiple weights included in the kernel, and can send the selected input data to the executor 150. By selecting the input data for the convolution operation, the sending efficiency and operating speed of the executor 150 can be improved.

[0054] Information about the zero values ​​of the weights stored in transmitter 140 can be referred to as "one-dimensional (1D) zero-value information". 1D zero-value information may include information about the relative position of the kernel's zero values ​​with respect to the input data in the 1D address system of reuse buffer 130.

[0055] Transmitter 140 can transmit the input data required for output data while incrementing a transmit counter. The transmitted input data can be referred to as "transmit input data 102," "transmit input data 103," and "transmit input data 104." The address of the buffer to be read by transmitter 140 can be pre-calculated. The address of the reuse buffer 130 storing transmit input data 102, transmit input data 103, and transmit input data 104 can be calculated based on the transmit counter. Transmit input data 102, transmit input data 103, and transmit input data 104 can overlap each other.

[0056] For example, when at least one transmitter is used, each of the at least one transmitter may correspond to each of transmit input data 102, transmit input data 103, and transmit input data 104 for outputting corresponding output data. In this example, at least one transmitter may share data stored in reuse buffer 130.

[0057] For example, in 1D zero-value information 172, the position indicated by "1" can be a position with a weight of zero value, and the position indicated by "0" can be a position with a weight of non-zero value. 1D zero-value information 172 can correspond to 2D zero-value information 171. The first three "1"s from bottom to top in 1D zero-value information 172 can correspond to a, b, and c in 2D zero-value information 171, and the last "1" in 1D zero-value information 172 can correspond to i in 2D zero-value information 171.

[0058] The transmitter 140 may skip positions indicated by "1" and may send input data corresponding to positions indicated by "0" stored in the reuse buffer 130 to the actuator 150. In response to each convolution operation, the transmitter 140 may send each of the transmission input data 102, transmission input data 103, and transmission input data 104 corresponding to positions indicated by "0" to the actuator 150.

[0059] The executor 150 can perform convolution operations in parallel. For example, when the acquirer 100 includes "N" transmitters, the executor 150 can perform convolution operations in parallel on the input data input from the "N" transmitters. The output data can be stored in a storage device (not shown). The storage device may be referred to as a "storage cell".

[0060] In one example, the convolution operation device can be applied to a rectangular kernel with weights of zero. The kernel may have a rectangular shape, and a portion of the weights included in the kernel may have zero values.

[0061] In another example, the kernel can have a shape other than a rectangle. In this example, the zero value can be assumed to be the value of the weight at a position in the smallest rectangle that overlaps with the kernel but does not overlap with it.

[0062] In another example, the kernel may include a kernel from which a portion of the weights have been removed by pruning. In this example, the zero value may be assumed to be the value of the weight at a position in the smallest rectangle that overlaps with the kernel but does not overlap with it.

[0063] Therefore, convolution operation devices can avoid unnecessarily loading or sending input data. Convolution operation devices can accelerate computation and reduce power consumption by reducing the number of load and send operations. Consequently, convolution operation devices can save space in the reuse buffer.

[0064] Figure 2 An example of the operation of the data transmission method used for convolution operations is shown.

[0065] In one example, the getter (e.g., Figure 1 The acquirer 100 may include a loader ( Figure 1 The loader 110), and at least one transmitter (e.g., Figure 1 The transmitter 140), buffer controller (not shown), and reuse buffer ( Figure 1 (Reusable buffer 130).

[0066] In operation 201, the loader can load the input data of the input feature map stored in memory according to the loading order, based on the input data stored in the reuse buffer, the 2D zero-value information of the shape of the kernel used for convolution operation and the weights of the kernel.

[0067] The loader can select the location of weights with non-zero values ​​based on the kernel shape and 2D zero-value information. The loader can then select input data from the input feature map corresponding to the location of the non-zero-valued weights that does not overlap with the input data stored in the reuse buffer. The loader can then load the selected input data.

[0068] In one example, the kernel may have a rectangular shape, and the 2D zero-value information may include 2D positional information indicating the location of weights with zero values ​​among the weights. In another example, the kernel may have a shape other than a rectangle, and the 2D zero-value information may include 2D positional information indicating the location of weights that do not overlap with the kernel within the smallest rectangle that overlaps with the kernel. In yet another example, the kernel may have a rectangular shape, and the 2D zero-value information may include 2D positional information indicating the location of weights that have been removed by pruning among the weights.

[0069] In operation 203, the buffer controller may store the loaded input data in a reuse buffer whose address is cyclically allocated according to the loading order.

[0070] In operation 205, each of at least one transmitter may select input data corresponding to each output data of the convolution operation from the input data stored in the reuse buffer based on the 1D zero-value information of the weights, and may send the selected input data to the executor.

[0071] The transmitter can select the position of a weight with a non-zero value from the weights based on 1D zero-value information. The transmitter can also select the input data corresponding to the position of the weight with a non-zero value from the input data stored in the reuse buffer. The transmitter can then send the selected input data to the actuator.

[0072] Figure 3 An example of a data transmission method used for convolution operations is shown.

[0073] Figure 3 Show the acquirer (e.g., Figure 1 Example of an operation performed by the acquirer 100 for each loop of data sent for the convolution operation.

[0074] In operation 301, it is determined whether the operation of the acquirer has been completed. If it is determined in operation 301 that the operation of the acquirer has not been completed, operation 302 can be executed.

[0075] In operation 302, the transmitter (e.g., Figure 1 The transmitter (140) can check whether the transmission is feasible. The transmitter can check the executor (e.g., Figure 1 The status of the actuator 150) can be checked, and the feasibility of the transmitter's transmission can be determined based on the results of the check.

[0076] In operation 303, the transmitter may select the next input data to be transmitted based on 1D zero-value information. The transmitter may use the 1D zero-value information stored in a reuse buffer (e.g., ...) Figure 1 The transmitter selects the input data from the input data in the reuse buffer 130) that corresponds to the position with a weight that has a non-zero value. The transmitter can then send the selected input data to the actuator.

[0077] In operation 304, it can be determined whether the reuse buffer is full. For example, operation 305 can be performed when the reuse buffer has space for storage.

[0078] In operation 305, the loader (e.g., Figure 1 The loader 110 can select the input data to be loaded next based on 2D zero-value information. The loader can select input data based on the shape of the kernel and the positional information of weights with zero values. The loader can select input data that does not overlap with the input data stored in the reuse buffer. The loader can store the selected input data in the reuse buffer.

[0079] In operation 306, the next loop can be executed.

[0080] Figure 4 This shows an example of input data loaded by a convolution operation device.

[0081] Kernel 411, input feature map 421, and output feature map 431 correspond to the general case without considering zero skipping. Kernel 412, input feature map 422, and output feature map 432 correspond to an example where the convolution operation device performs zero skipping.

[0082] All weights from 1 to 9 in kernel 411 can have non-zero values. Each output value of output feature map 431 can be output through a convolution operation between kernel 411 and input feature map 421. For example, the output value 0 can be output through a convolution operation between kernel 411 and the input values ​​from 0 to 8 corresponding to kernel 411. When kernel 411 slides down by one step, the output value 1 can be output through a convolution operation between kernel 411 and the input values ​​from 3 to 11 corresponding to kernel 411.

[0083] In kernel 412, weights 1 through 3 and 9 can have zero values, while weights 4 through 8 can have non-zero values. Each output value of output feature map 432 can be output through a convolution operation between kernel 412 and input feature map 422. For example, the output value 0 can be output through a convolution operation between kernel 412 and the input values ​​0 through 4 corresponding to the positions of the weights with non-zero values ​​in kernel 412. When kernel 412 slides down by one step, the output value 1 can be output through a convolution operation between kernel 412 and the input values ​​3 through 7 corresponding to the positions of the weights with non-zero values ​​in kernel 412.

[0084] As mentioned above, convolution operation devices can avoid unnecessarily loading or sending input data. Convolution operation devices can accelerate computation and reduce power consumption by reducing the number of loads and sends performed. Convolution operation devices can save space in reuse buffers.

[0085] Figure 5 Examples of 2D zero-value information and 1D zero-value information used by convolution operation devices are shown.

[0086] Reference Figure 5Kernel 510 may include weights with zero values. In kernel 510, the weights a, b, c, d, e, f, g, h, and i may have non-zero values, while other weights may have zero values. In one example, kernel 510 may have a rectangular shape, and a portion of the weights included in kernel 510 may have zero values. In another example, kernel 510 may have a shape other than a rectangle. In this example, zero values ​​may be assumed to be the values ​​of the weights at positions within the smallest rectangle that overlaps with kernel 510 but do not overlap with kernel 510. In another example, kernel 510 may include a kernel from which a portion of the weights has been removed by pruning. In this example, zero values ​​may be assumed to be the values ​​of the weights at positions within the smallest rectangle that overlaps with kernel 510 but do not overlap with kernel 510.

[0087] 2D zero-value information 520 (e.g., Figure 1 The 2D zero-value information 520 can be represented in 2D as the location of weights included in kernel 510. The 2D zero-value information 520 may include width and height, and indicates the locations of weights 521, 522, 523, 524, 525, and 526 with zero values. The loader (e.g., Figure 1 The loader 110 can store 2D zero-value information 520 and can selectively load input data of the input feature map based on the 2D zero-value information 520.

[0088] 1D zero-value information 530 (e.g., Figure 1 The 1D zero-value information (172) can be represented in 1D by the location of the weights included in the kernel 510. The transmitter (e.g., Figure 1 The transmitter 140 may need to select the storage location in the reuse buffer (e.g., Figure 1 Input data in the storage space of the 1D address system of the reuse buffer 130) is therefore more advantageous as 1D zero-value information 530 represented as 1D location than 2D zero-value information 520.

[0089] The 1D zero-value information 530 can indicate the positions of weights 531, 532, 533, 534, 535, and 536 that have zero values. Figure 5 In this context, "1" indicates that a weight has a zero value, and "0" indicates that a weight has a non-zero value. Weights 531, 532, 533, 534, 535, and 536 with zero values ​​can correspond to weights 521, 522, 523, 524, 525, and 526 with zero values, respectively. The transmitter can store 1D zero-value information 530 and can select, based on the 1D zero-value information 530, which will be sent to the actuator (e.g., ...). Figure 1 Input data for actuator 150.

[0090] Figure 6 An example of the acquirer configuration is shown.

[0091] Reference Figure 6 The acquirer 100 may include a loader 110, a buffer controller 120, a reuse buffer 130, and at least one transmitter (e.g., transmitter 141, transmitter 142, and transmitter 143).

[0092] The loader 110 can load input data of the input feature map stored in memory according to the loading order, based on the input data stored in the reuse buffer 130, the 2D zero-value information of the shape of the kernel to be used for convolution operation and the weights of the kernel.

[0093] The loader 110 can select the location of weights with non-zero values ​​among the weights based on the shape of the kernel and 2D zero-value information. The loader 110 can select input data from the input feature map corresponding to the location of the weight with non-zero values ​​that does not overlap with the input data stored in the reuse buffer 130. The loader 110 can load the selected input data.

[0094] The buffer controller 120 can store the loaded input data in a reuse buffer 130 that cyclically allocates addresses according to the loading order.

[0095] Each of transmitters 141, 142, and 143 may select input data corresponding to each output data of the convolution operation from the input data stored in the reuse buffer 130 based on the 1D zero-value information of the weights, and may send the selected input data to the executor.

[0096] Each of transmitters 141, 142, and 143 can select the position of a weight with a non-zero value from among the weights based on 1D zero-value information. Each of transmitters 141, 142, and 143 can select input data from the input data stored in the reuse buffer 130 corresponding to the position of the weight with a non-zero value. Each of transmitters 141, 142, and 143 can send the selected input data to the actuator.

[0097] In one example, the kernel may have a rectangular shape, and the 2D zero-value information may include 2D positional information indicating the location of weights with zero values ​​among the weights. In another example, the kernel may have a shape other than a rectangle, and the 2D zero-value information may include 2D positional information indicating the location of weights that do not overlap with the kernel within the smallest rectangle that overlaps with the kernel. In yet another example, the kernel may have a rectangular shape, and the 2D zero-value information may include 2D positional information indicating the location of weights that have been removed by pruning among the weights.

[0098] Figure 7 An example configuration of a convolution operation device is shown.

[0099] Reference Figure 7 The convolution operation device may include a memory 710, a acquirer 100, an executor 150, and a storage device 730. The acquirer 100 may include a loader 110, a buffer controller 120, a reuse buffer 130, and at least one transmitter (e.g., transmitter 141, transmitter 142, and transmitter 143).

[0100] The loader 110 can load the input data of the input feature map stored in the memory 710 according to the loading order, based on the input data stored in the reuse buffer 130, the 2D zero-value information of the shape of the kernel to be used for convolution operation and the weights of the kernel.

[0101] The buffer controller 120 can store the loaded input data in a reuse buffer 130 that cyclically allocates addresses according to the loading order.

[0102] Each of transmitters 141, 142, and 143 may select input data corresponding to each output data of the convolution operation from the input data stored in the reuse buffer 130 based on the 1D zero-value information of the weights, and may send the selected input data to the executor 150.

[0103] In this regard Figure 1 , Figure 6 and Figure 7The described devices, units, modules, apparatuses, and other components (such as acquirer 100, loader 110, buffer controller 120, reuse buffer 130, transmitter 140, transmitter 141, transmitter 142, transmitter 143, actuator 150, memory 710, and storage device 730) are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0104] Figures 2 to 5The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer) implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0105] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers, such as machines or special-purpose computers, to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0106] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0107] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be interpreted as included in the disclosure.

Claims

1. A data transmission method for a acquirer, the acquirer comprising a loader, at least one transmitter, a buffer controller, and a reuse buffer, the data transmission method comprising: The loader loads the input data of the input feature map stored in memory according to the loading order, based on the input data stored in the reuse buffer, the two-dimensional zero-value information of the shape of the kernel to be used for convolution operation and the weight of the kernel. The buffer controller stores the input data loaded by the loader in a reuse buffer whose address is cyclically allocated according to the loading order; and The input data corresponding to each output data of the convolution operation is selected from the input data stored in the reuse buffer by the buffer controller based on the weighted one-dimensional zero value information of each of the at least one transmitters, and the selected input data is output.

2. The data transmission method according to claim 1, wherein, The kernel has a rectangular shape, and Two-dimensional zero-value information includes two-dimensional position information indicating the location of one or more weights, each of which has a zero value.

3. The data transmission method according to claim 1, wherein, The kernel has shapes other than rectangular shapes, and Two-dimensional zero-value information includes two-dimensional positional information indicating the location of one or more weights that do not overlap with the kernel within the smallest rectangle that overlaps with the kernel.

4. The data transmission method according to claim 1, wherein, The kernel has a rectangular shape, and Two-dimensional zero-value information includes two-dimensional location information indicating the location of one or more weights that have been removed through pruning.

5. The data transmission method according to any one of claims 1 to 4, wherein, The steps for loading input data include: The position of weights with non-zero values ​​is selected based on the shape of the kernel and two-dimensional zero-value information. Select input data from the input feature maps corresponding to positions with non-zero weights that does not overlap with input data stored in the reuse buffer; and Load the selected input data.

6. The data transmission method according to any one of claims 1 to 4, wherein, The steps for selecting input data include: Based on one-dimensional zero-value information, select the position of weights with non-zero values ​​among the weights; Select the input data corresponding to the position with a non-zero weight from the input data stored in the reuse buffer; and The selected input data is sent to the executor.

7. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the data transmission method according to any one of claims 1 to 6.

8. A acquirer, comprising: Loader; At least one transmitter; Buffer controller; and Reuse buffer, The loader is configured to load input data of the input feature maps stored in memory according to the loading order, based on the input data stored in the reuse buffer, the two-dimensional zero-value information of the shape of the kernel to be used for convolution operations, and the weights of the kernel. The buffer controller is configured to store the input data loaded by the loader in a reuse buffer whose addresses are cyclically allocated according to the loading order, and Each of the at least one transmitter is configured to: select input data corresponding to each output data of the convolution operation from the input data stored in the reuse buffer by the buffer controller based on the weighted one-dimensional zero-value information, and output the selected input data.

9. The acquirer according to claim 8, wherein, The kernel has a rectangular shape, and Two-dimensional zero-value information includes two-dimensional position information indicating the location of one or more weights, each of which has a zero value.

10. The acquirer according to claim 8, wherein, The kernel has shapes other than rectangular shapes, and Two-dimensional zero-value information includes two-dimensional positional information indicating the location of one or more weights that do not overlap with the kernel within the smallest rectangle that overlaps with the kernel.

11. The acquirer according to claim 8, wherein, The kernel has a rectangular shape, and Two-dimensional zero-value information includes two-dimensional location information indicating the location of one or more weights that have been removed through pruning.

12. The acquirer according to claim 8, wherein, The loader is configured as follows: The position of weights with non-zero values ​​is selected based on the shape of the kernel and two-dimensional zero-value information; Select input data from the input feature map corresponding to the position with non-zero weights that does not overlap with the input data stored in the reuse buffer; and Load the selected input data.

13. The acquirer according to claim 8, wherein, The at least one transmitter is configured to: The position of weights with non-zero values ​​is selected based on one-dimensional zero-value information; Select the input data corresponding to the position with a non-zero weight from the input data stored in the reuse buffer; and The selected input data is sent to the executor.

14. The acquirer according to any one of claims 8 to 13, further comprising: The memory is configured to store the input feature maps; and An executor is configured to perform parallel convolution operations on selected input data output from the at least one transmitter.

15. A acquirer, comprising: One or more processors are configured as follows: Input data of the input feature map stored in memory is loaded by loading the feature values ​​of the input feature map corresponding to the positions of the non-zero values ​​of the weights of the kernel to be used for the convolution operation and skipping the feature values ​​of the input feature map corresponding to the positions of the zero values ​​of the weights of the kernel. Store the loaded input data in a reuse buffer; and The kernel uses one-dimensional zero-value information based on its weights to select a portion of the input data to be output, stored in a reuse buffer.

16. The acquirer according to claim 15, wherein, In the case where the kernel has a non-rectangular shape, the one or more processors are configured to assign zero values ​​to any weights that do not overlap with the kernel within the smallest rectangle that completely contains the kernel.

Images