Low latency convolution operation apparatus and method for mobile environment
The low-latency convolution operation device addresses the inefficiencies of existing methods by processing convolution operations with a matrix operation program that matches input and filter matrices based on convolution parameters, resulting in reduced latency and memory usage, enhancing neural network performance in mobile environments.
Patent Information
- Application Number
- PCT/KR2023/015677
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2023-10-12
- Publication Date
- 2024-11-21
AI Technical Summary
Existing convolution operation methods in mobile environments, such as General Matrix Multiplication (GEMM), suffer from high latency and inefficiency due to data replication and movement overhead, as well as low data reuse rates, which are exacerbated by the inability to process small amounts of data quickly and efficiently.
A low-latency convolution operation device and method that includes a processor executing a matrix operation program, which generates output matrices by matching input and filter matrices considering convolution parameters like stride and dilation, and processes input and filter blocks to reduce memory usage and latency, allowing for efficient neural network processing in mobile environments.
This approach reduces memory usage and operation latency, enhances processing performance, and decreases energy consumption, enabling faster and more efficient artificial neural network services in mobile systems by eliminating additional overhead and improving computational efficiency.
Smart Images

Figure KR2023015677_21112024_PF_FP_ABST
Abstract
Description
Low-latency convolutional computing device and method for mobile environments
[0001] The present invention relates to a low-latency convolution operation device and method for a mobile environment.
[0002] Many artificial neural network-based services, such as object recognition and image processing, currently rely on convolution operations as a core component. Most existing artificial neural network implementation solutions utilize techniques that convert convolution operations into general matrix multiplication (GEMM) operations for processing. This applies even in mobile environments.
[0003] However, the process of converting convolutional operations to GEMM operations incurs data replication and movement overhead, lowering data reuse, and degrading computational performance. In existing server or desktop environments, networks are often trained based on large amounts of data, or multiple requests for neural network services are processed simultaneously. In such environments, a strategy is adopted to collate and process large amounts of data simultaneously through a single artificial neural network operation. This reduces the time consumed by data replication and movement overhead in the overall computation, and increases data reuse by repeating the same operation on a large amount of data.
[0004] However, unlike server or desktop environments, in mobile environments, most neural network services are provided for a single piece of data based on individual user requests. This means that strategies for simultaneous collection and processing of large amounts of data are impossible. Consequently, the performance degradation caused by data duplication and movement overhead and low data reuse rates inherent in existing GEMM methods are directly exposed.
[0005] Furthermore, it's crucial for artificial neural network services offered in mobile environments to maximize user satisfaction by delivering neural network results tailored to individual needs as quickly as possible. To achieve this, minimizing the time from user request to result delivery is essential, even for small amounts of data.
[0006] Therefore, the existing GEMM method processes large amounts of data simultaneously, increasing average throughput per data. However, this increases latency between each data point and the output of the results. Therefore, the existing GEMM method is not suitable for mobile environments, where it is crucial to provide results as quickly as possible from user requests.
[0007] In this regard, Fig. 1 is a diagram illustrating the iteration of N dimensions in a general GEMM operation. The N-dimensional iteration (N Iteration) reuses the data of matrix A. Fig. 2 is a diagram illustrating the iteration of the output width (Wout) dimension in a general 2D CNN operation. The iteration (Wout Iteration) of the output width dimension (Wout) reuses the data of the filter tensor.
[0008] For example, as illustrated in Figure 1, the matrix multiplication (GEMM) operation iterates over three operation dimensions of M, N, and K for three matrices A, B, and C in the form of a nested loop. At this time, the loop iteration for each operation dimension of M, N, and K reuses the data corresponding to each matrix B, A, and C in the next operation. On the other hand, the 2D CNN operation iterates over a total of seven dimensions of the four-dimensional tensors of input, filter, and output in the form of a nested loop. At this time, the operation dimensions are Batch, Input Channels (Cin), Output Channels (Cout), Output Height (Hout), Output Width (Wout), Filter Height (Hfil), and Filter Width (Wfil).
[0009] The method of converting a conventional CNN to a GEMM mainly maps the input, filter, and output tensors to matrices B, A, and C, respectively, and maps the seven CNN operation dimensions to three GEMM operation dimensions. In this case, the CNN operation dimensions except for the filter spatial dimension (Hfil, Wfil) can be directly mapped to the GEMM operation dimensions. However, the filter spatial dimension has different data access and reuse patterns from the GEMM operation dimensions M, N, and K, so direct mapping is impossible. To solve this, additional processing such as Im2col is required, which increases the operation time due to overhead such as data duplication and rearrangement.
[0010] Therefore, the method of converting existing CNNs to GEMMs is inefficient because the spatial dimension of the filter cannot be expressed in terms of the computational dimension of GEMMs.
[0011] The present invention is intended to solve the aforementioned problems, and proposes a low-latency convolution operation device and method for a mobile environment that provides fast processing performance for a small amount of data in a mobile environment compared to the existing GEMM method.
[0012] However, the technical task that this embodiment seeks to achieve is not limited to the technical task described above, and other technical tasks may exist.
[0013] As a technical means for solving the above-described technical problem, a low-latency convolution operation device according to an embodiment of the present invention includes a memory in which a matrix operation program is stored; and a processor for executing the program stored in the memory, wherein the matrix operation program includes a matrix operation unit for generating an output matrix according to a matrix operation for each element of an input matrix and a filter matrix, wherein the matrix operation unit includes a step (a) of matching rows of an input matrix with columns of a filter matrix by considering convolution parameters including a stride and a dilation, a step (b) of generating an output block by operating a filter block moving in a depth direction with respect to a width dimension of a filter tensor and an input block moving in a horizontal direction, and a step (c) of generating an output matrix by summing all output blocks produced by repeating steps (a) and (b) into one matrix, wherein the input matrix includes input blocks in which the width dimension and the channel dimension are listed in the horizontal direction and the vertical direction, respectively, and the filter matrix includes filter blocks in which the channel dimension is listed in the horizontal direction and the vertical direction, and the width dimension is listed in the depth direction.
[0014] A low-latency convolution operation method performed by a low-latency convolution operation device according to another embodiment of the present invention includes a step of generating an output matrix by a matrix operation unit according to a matrix operation for each element of an input matrix and a filter matrix, wherein the generating step includes (a) a step of matching rows of an input matrix with columns of a filter matrix by the matrix operation unit in consideration of convolution parameters including a stride and a dilation, (b) a step of generating an output block by operating a filter block moving in a depth direction and an input block moving in a horizontal direction with respect to a width dimension of a filter tensor, and (c) a step of generating an output matrix by summing all output blocks produced by repeating steps (a) and (b) into one matrix, wherein the input matrix includes input blocks in which the width dimension and the channel dimension are listed in the horizontal direction and the vertical direction, respectively, and the filter matrix includes filter blocks in which the channel dimension is listed in the horizontal direction and the vertical direction, and the width dimension is listed in the depth direction.
[0015] Any of the aforementioned solutions to the present invention can reduce memory usage and computational latency during artificial neural network processing compared to existing methods. This enables more efficient artificial neural network processing in mobile environments, utilizing limited resources such as memory, hardware performance, and battery life.
[0016] Furthermore, reduced memory usage offers benefits in processing performance. Furthermore, memory access bottlenecks are reduced in mobile systems, including main memory and buses, where various computational resources, such as CPUs, GPUs, and DSPs, share common resources. This allows users to enjoy seamless artificial neural network services.
[0017] Additionally, latency is reduced by eliminating additional overhead and increasing computational efficiency. This reduces the amount of energy required to operate artificial neural networks, making it easier for mobile systems with limited battery capacity to utilize artificial neural network-based services.
[0018] Figure 1 is a diagram illustrating the iteration of N dimensions in a general GEMM operation.
[0019] Figure 2 is a diagram illustrating the iteration of the output width (Wout) dimension in a typical 2D CNN operation.
[0020] FIG. 3 is a diagram illustrating the structure of a low-latency convolution operation device for a mobile environment according to one embodiment of the present invention.
[0021] FIG. 4 is a drawing illustrating the structure of a matrix operation unit according to one embodiment of the present invention.
[0022] Figure 5 is a flowchart explaining the operation method of the matrix operation unit of Figure 4.
[0023] Figure 6 is a drawing explaining the computational method of a structure combining Im2Col and GEMM among the methods for converting existing CNNs into GEMMs.
[0024] Figure 7 is a drawing for explaining the operation method of the matrix operation unit of Figure 4.
[0025] Figure 8 illustrates an algorithm of a low-latency convolutional neural network operation method according to one embodiment of the present invention.
[0026] Figure 9 illustrates the results of comparing a low-delay convolution operation method according to one embodiment of the present invention with a conventional method.
[0027] Below, with reference to the attached drawings, embodiments of the present invention are described in detail to facilitate easy implementation by those skilled in the art. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity, and similar reference numerals have been used throughout the specification to indicate similar elements.
[0028] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.
[0029] In this specification, the term 'unit' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be realized by using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, the '~ unit' is not limited to software or hardware, and the '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, the '~ unit' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and '~sub-units' may be combined into a smaller number of components and '~sub-units' or further separated into additional components and '~sub-units'. Furthermore, the components and '~sub-units' may be implemented to recycle one or more CPUs within the device.
[0030] A network is a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), the Internet (WWW), wired and wireless data communication networks, telephone networks, and wired and wireless television communication networks. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC), LiFi, etc.
[0031] Hereinafter, one embodiment of the present invention will be described in detail with reference to the attached drawings.
[0032] First, in existing mobile environments, matrix multiplication (GEMM) operations are primarily used to process convolutional operations. However, as described above with reference to Figures 1 and 2, GEMM operations suffer from inefficiencies such as slow processing speeds due to the overhead of data replication and relocation, and lower data reuse rates compared to convolutional operations. In particular, these inefficiencies lead to even greater performance degradation in AI neural networks in mobile environments.
[0033] In order to solve the aforementioned problem, the present invention relates to a low-latency convolution operation device and method for accelerating convolution neural network processing, which is a core component in artificial neural network processing, by taking into account the characteristics of a mobile environment.
[0034] FIG. 3 is a diagram illustrating the structure of a low-latency convolution operation device for a mobile environment according to one embodiment of the present invention.
[0035] Referring to FIG. 3, the convolution operation device (100) may include a communication module (110), a memory (120), a processor (130), and a database (140).
[0036] The communication module (110) can receive input data and transmit it to the processor (130). Here, the communication module (110) can be a device including hardware and software necessary to transmit and receive signals such as control signals or data signals through a wired or wireless connection with another network device.
[0037] The memory (120) may have a matrix operation program recorded therein. The matrix operation program includes a matrix operation unit that generates an output matrix according to matrix operations for each element of an input matrix and a filter matrix. At this time, the matrix operation unit includes a step (S110) of matching rows of an input matrix with columns of a filter matrix by considering convolution parameters including stride and dilation, a step (S120) of generating an output block by operating a filter block moving in the depth direction and an input block moving in the horizontal direction with respect to the width dimension of a filter tensor, and a step (S130) of generating an output matrix by adding all output blocks produced by repeating steps S110 and S120 into one matrix.
[0038] The memory (120) may include magnetic storage media or flash storage media in addition to volatile storage devices that require power to maintain stored information, but the scope of the present invention is not limited thereto.
[0039] The memory (120) may store a separate program, such as an operating system for processing and controlling the processor (130), or may perform a function for temporarily storing input or output data.
[0040] The processor (130) executes a matrix operation program (hereinafter, “program”) stored in the memory (120) and provides a function of controlling the hardware of the convolution operation device (100) according to the execution of the program. That is, the processor (130) can perform hardware control functions such as a necessary file system, memory allocation, network, basic library, timer, device control (display, media, input device, 3D, etc.), and other utilities according to the execution of the program.
[0041] In addition, each specific step of the matrix operation process according to the execution of the program will be described later with reference to FIGS. 4 to 7.
[0042] The processor (130) may include any type of device capable of processing data. For example, it may refer to a hardware-embedded data processing device having a physically structured circuit to perform a function expressed by a code or command included in a program. Examples of such hardware-embedded data processing devices include processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.
[0043] The database (140) stores or provides data required for the convolution operation device (100) under the control of the processor (130). This database (140) may be included as a separate component from the memory (120), or may be built in a portion of the memory (120).
[0044] FIG. 4 is a drawing illustrating the structure of a matrix operation unit according to one embodiment of the present invention, and FIG. 5 is a flowchart illustrating an operation method of the matrix operation unit of FIG. 4.
[0045] Referring to FIG. 4, the processor (130) includes a matrix operation unit (150) that generates an output matrix (400) according to matrix operations for each element of the input matrix (200) and the filter matrix (300).
[0046] Referring to FIG. 5, the matrix operation unit (150) includes a step (S110) of matching rows of an input matrix (200) with columns of a filter matrix (300) by considering convolution parameters including stride and dilation, a step (S120) of generating an output block by calculating a filter block moving in the depth direction and an input block moving in the horizontal direction with respect to the width dimension of a filter tensor, and a step (S130) of generating an output matrix (400) by adding all output blocks produced by repeating steps S110 and S120 into one matrix. At this time, the convolution parameters include, but are not limited to, stride and dilation.
[0047] Before step S110, the matrix operation unit (150) can map the convolution operation dimension for the target calculation to the GEMM operation dimension, and tile the input matrix (200) and the filter matrix (300) into a plurality of blocks of a predetermined size. This tiling process uses the existing GEMM operation method, and a detailed description thereof is omitted. Here, a block means a sub-matrix obtained by dividing each matrix into a predetermined size, and the sub-matrix includes elements that are adjacent to each other.
[0048] Here, the input matrix (200) includes blocks in which the width dimension and the channel dimension are listed in the horizontal and vertical directions, respectively. In addition, the filter matrix (300) includes filter blocks in which the channel dimension is listed in the horizontal and vertical directions, and the width dimension is listed in the depth direction. The output matrix (300) includes blocks in which the width dimension and the channel dimension are listed in the horizontal and vertical directions, respectively.
[0049] Therefore, the present invention proposes a novel algorithm that eliminates the inefficiencies inherent in existing GEMM methods. This significantly reduces artificial neural network processing time in mobile environments.
[0050] Specifically, the present invention proposes a new algorithm that addresses the problem of converting existing CNN operations into GEMM operations, which stems from the limitation that GEMM operations cannot express all dimensions of CNN operations. That is, a low-latency convolutional neural network operation device according to an embodiment of the present invention extends the existing method by adding an operation dimension that has the same characteristics as the filter space dimension of CNN operations that cannot be expressed in the existing GEMM algorithm.
[0051] Here, the added operation dimension is a dimension that expresses the spatial dimension (Spatial dimension (Width, Height)) of the filter (kernel) of the CNN operation. This dimension has the characteristic of reusing input (feature map) data and output data and newly accessing the filter data. Meanwhile, the present invention can process the operation using the existing GEMM algorithm as it is for the part of the dimension of the CNN operation that can be processed by the GEMM algorithm. However, the present invention can process the part (spatial dimension of the filter) that cannot be processed by the existing GEMM algorithm by using the newly added operation dimension. That is, the present invention can efficiently process all CNN operations, including the spatial dimension of the filter that could not be processed by the existing GEMM algorithm, through the matrix operation unit (150). Therefore, convolutional neural network processing in a mobile environment can be performed quickly and efficiently.
[0052] Fig. 6 is a drawing explaining an operation method of a structure combining Im2Col and GEMM among methods for converting existing CNNs into GEMMs, and Fig. 7 is a drawing explaining an operation method of the matrix operation unit of Fig. 4.
[0053] First, the problems that occur when performing operations according to the existing combination structure (Im2Col+GEMM) will be explained with reference to Fig. 6.
[0054] Figure 6 is a process of performing CNN operations using the existing combination structure (Im2Col+GEMM), which is the most commonly used method of converting CNN to GEMM.
[0055] In the existing combination structure (Im2Col+GEMM), the filter width dimension (Wf) is flattened to K matrix dimensions using the input channel dimension (Ci). Accordingly, as illustrated in Fig. 6(a), the reuse of input blocks in the filter width dimension (Wf) is expressed as duplication and shift of the input blocks. In other words, the existing combination structure goes through a process of duplicating and rearranging the input data (such as Im2Col) to process the inexpressible filter space dimension operation of CNN within the framework of the GEMM operation. In this process, the data reuse opportunities provided by the filter space dimension of CNN are limited.
[0056] However, the low-latency convolution operation device (100) of the present invention can process operations without additional processing such as data duplication and rearrangement, unlike the existing combination structure, through iteration for the filter width dimension (Wf) added to the filter matrix (300).
[0057] Hereinafter, the operation process according to the matrix operation unit (150) of the present invention will be described with reference to FIG. 7. At this time, FIGS. 7(a) to 7(e) illustrate operation processes that are mathematically identical to the operation processes illustrated in FIGS. 6(a) to 6(e), respectively.
[0058] Referring to FIG. 4 and FIG. 7(a), the matrix operation unit (150) can generate an output matrix (400) according to matrix operations for each element of the input matrix (200) and the filter matrix (300).
[0059] For example, the matrix operation unit (150) can produce an output block based on the horizontal position of the input block and the depth position of the filter block, according to Equation 1.
[0060] <Formula 1>
[0061] Wo+Wf-1
[0062] Here, Wo is the output width dimension, and Wf is the filter width dimension.
[0063] The matrix operation unit (150) can match the rows of the input matrix (200) with the columns of the filter matrix (300) with respect to the input channel dimension (Ci). For example, as illustrated in FIG. 7(a), the matrix operation unit (150) can perform operations between the filter blocks (301-303) in which the input channel dimension is located in the first column (Ci=0) and the input blocks (201-207) in which the input channel dimension is located in the first row (Ci=0).
[0064] In addition, the matrix operation unit (150) can generate output blocks (401-405) by operating filter blocks (301-303, 311-313) moving in the depth direction with respect to the filter width dimension (Wf) and input blocks (201-207, 211-217) moving in the horizontal direction. For example, as illustrated in FIGS. 7(b) to 7(d), the input channel dimension can be set to the first column of the filter matrix and the first row (Ci=0) of the input matrix. At this time, when the filter blocks (301, 302, 303) move to 0 depth (Wf=0), 1 depth (Wf=1), and 2 depths (Wf=2) of the filter width dimension, the input blocks (201-205, 202-206, 203-207) can perform operations while moving one block at a time in the horizontal direction. At this time, the number of columns of the input block operated with one filter block is 5, which can be determined by the number of columns of the output block.
[0065] Next, referring to FIG. 7(e), the matrix operation unit (150) can perform operations on the filter blocks (311-313) and the input blocks (211-217) by moving one block at a time with respect to the filter width dimension (Wf) even when the input channel dimension Ci = 1 (the second column of the filter matrix and the second row of the input matrix). That is, a multiplication operation is performed between the elements of the input blocks (201-207, 211-217) and the filter blocks (301-303, 311-313), and all of the calculated results are added together to generate the output blocks (401-405).
[0066] At this time, in the operation process illustrated in Figures 7(a) to 7(e), the opaque block represents data loaded into the register, and the arrows illustrated from input to output mean that a MAC operation is performed on the loaded data.
[0067] For example, referring to FIGS. 7(b) to 7(d), when the input channel dimension is Ci=0, a process in which the matrix operation unit (150) performs an operation by moving one block at a time with respect to the filter width dimension (Wf) is described.
[0068] As illustrated in Fig. 7(b), the first filter block (301) located at 0 depth (Wf=0) performs operations with the input blocks (201-205) of the first group, and the results of the operations can be stored in the locations of the first output block (401) to the fifth output block (405).
[0069] As illustrated in Fig. 7(c), the second filter block (302) located at a depth of 1 (wf=1) performs operations with the input blocks (202-206) of the second group, and the results of the operations can be stored in the locations of the first output block (401) to the fifth output block (405).
[0070] As illustrated in Fig. 7(d), the third filter block (303) located at a depth of 2 (wf=2) performs an operation with the input blocks (203-207) of the third group, and the result of the operation can be stored in the locations of the first output block (401) to the fifth output block (405).
[0071] In addition, as illustrated in Fig. 7(e), even when the input channel dimension is Ci = 1, the matrix operation unit (150) can perform the operation by moving one block at a time with respect to the filter width dimension. Similarly, the operation result values of each filter block (311, 312, 313) and the corresponding group-specific input blocks (211-215, 212-216, 213-217) can be stored in the locations of the first output block (401) to the fifth output block (405). Finally, the accumulated stored operation result values are summed, and the first output block (401) to the fifth output block (405) can be generated.
[0072] Additionally, the operation result values that are batch-processed for each input group (five columns) can be stored at once in the output group (five columns) including the first output block (401) to the fifth output block (405). For example, the output group (401-405) can be located in the first row (Wo=0) of the output width dimension.
[0073] For example, the direction of the arrow (MAC operation) can indicate in which position of the output block the operation result of each group input block for the filter width dimension (Wf) is stored.
[0074] For example, according to Equation 1, for the first group, Wo(0)+Wf(0)-1=-1, for the second group, Wo(0)+Wf(1)-1=0, and for the third group, Wo(0)+Wf(2)-1=1 can be derived. That is, the arrow direction (↘) connecting the input block (201-205) and the output group (401-405) of the first group can point to the right (-1). The arrow direction (↓) connecting the input block (202-205) and the output group (401-405) of the second group can remain in place (0). The arrow direction (↙) connecting the input block (203-207) and the output group (401-405) of the third group can point to the left (1).
[0075] That is, the present invention can omit the process of duplicating and rearranging the input blocks of the input matrix (200) by using the filter width dimension (Wf) of the filter matrix (300). In addition, the data reuse opportunity of the filter space dimension of CNN can be maximized. In this way, the present invention can omit the data duplication process while performing mathematically identical operations to the existing combination structure. In other words, the same data (input blocks) can be reused for more operations and the operations can be performed. As a result, compared to the operations of the existing combination structure, it uses less memory and reduces the number of data accesses, enabling faster operations.
[0076] Therefore, the present invention is a new algorithm that adds an operation dimension (filter width dimension (Wf)) that expresses the filter space dimension of CNN that cannot be expressed by existing GEMM operations. In other words, it can provide the effect of solving the inefficiency problem of the method of processing existing CNNs by mapping them to GEMM operations through the filter width dimension (Wf).
[0077] Figure 8 illustrates an algorithm of a low-latency convolutional neural network operation method according to one embodiment of the present invention.
[0078] The present invention is a method for improving the processing of artificial neural network convolution operations based on the existing matrix multiplication (GEMM), and aims to propose a convolution operation algorithm optimized for the characteristics of a mobile environment by adding a new operation dimension to the existing GEMM method algorithm. To demonstrate this, the algorithm was implemented on a processor of the ARMv8 architecture used by most mobile systems in the manner illustrated in Fig. 8. The inner kernel is a key component for implementing the convolution operation, and is ARMv8-A assembly code for performing the operation as illustrated in Fig. 8. This code utilizes various hardware acceleration technologies such as ARM NEON and prefetching to perform the operation in the most efficient manner according to fixed output width (wo) and output channel (co) values.
[0079] In the algorithm of Figure 8, lines 7 through 13 correspond to the inner kernel, and the entire operation is implemented using a tiling method. Considering the workload's input parameters, the inner kernel with the optimally efficient combination of wo and co values is selected, and this is repeatedly performed to configure the entire operation.
[0080] To achieve this efficiently, lines 1-6 appropriately split the input parameters and rearrange their order to maximize data reuse within the given hardware's memory hierarchy. The cout parameter is additionally split into coo as well as co, and the output tensor values are streamed from the register level, the filter values are streamed from the L1 cache, and the input values are streamed from the L2 / Last level cache, and are arranged so that they can be shared and reused across multiple cores.
[0081] In addition, to enable sequential data access according to the wo and co values, the input and output data were rearranged in the order of N-(C / co)-HW-co, and the filter data was rearranged in the order of (K / co)-RCS-co.
[0082] Figure 9 illustrates the results of comparing a low-delay convolution operation method according to one embodiment of the present invention with a conventional method.
[0083] Referring to FIG. 9, the operation code of mGEMM implemented in a manner according to one embodiment of the present invention was compared with existing methods, and as a result, performance improvement was observed on various mobile devices for the YoloV3-Tiny Object Detection task.
[0084] That is, the present invention implements mGEMM as C-based software, and as a result of comparing its performance with existing methods in an ARMv8-based mobile environment, it was confirmed that there was a performance improvement of 1.29 to 1.58 times compared to existing methods.
[0085] An embodiment of the present invention may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Computer-readable media may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.
[0086] Although the methods and devices of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.
[0087] The above description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.
[0088] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.
Claims
1. In a low-latency convolution operation device, Memory where matrix operation programs are stored; and A processor that executes a program stored in the above memory, The above matrix operation program is, It includes a matrix operation unit that generates an output matrix according to matrix operations for each element of the input matrix and the filter matrix. The above matrix operation unit includes a step (a) of matching rows of the input matrix with columns of the filter matrix by considering convolution parameters including stride and dilation, a step (b) of generating an output block by calculating a filter block moving in the depth direction and an input block moving in the horizontal direction with respect to the width dimension of the filter tensor, and a step (c) of generating the output matrix by adding all output blocks produced by repeating steps (a) and (b) into one matrix. The above input matrix includes the input blocks whose width dimension and channel dimension are listed in the horizontal and vertical directions, respectively, A low-latency convolution operation unit, wherein the filter matrix includes filter blocks whose channel dimensions are listed in the horizontal and vertical directions and whose width dimension is listed in the depth direction.
2. In paragraph 1, The above matrix operation unit A low-latency convolution operation device that maps a convolution operation dimension for a target calculation to a GEMM operation dimension and tiles the input matrix and filter matrix into a plurality of blocks of a predetermined size.
3. In paragraph 1, The above matrix operation unit A low-latency convolution operation device that produces the output block based on the horizontal position of the input block and the depth position of the filter block according to Equation 1. <Formula 1> Wo+Wf-1 Here, Wo is the output width dimension, and Wf is the filter width dimension.
4. In a low-latency convolution operation method performed by a low-latency convolution operation device, A step of generating an output matrix according to a matrix operation for each element of an input matrix and a filter matrix by a matrix operation unit, The above creation steps are (a) the matrix operation unit matches the rows of the input matrix with the columns of the filter matrix by considering convolution parameters including stride and dilation; (b) a step of generating an output block by calculating a filter block moving in the depth direction and an input block moving in the horizontal direction with respect to the width dimension of the filter tensor, and (c) a step of generating the output matrix by adding all output blocks produced by repeating the above steps (a) and (b) into one matrix, The above input matrix includes the input blocks whose width dimension and channel dimension are listed in the horizontal and vertical directions, respectively, A low-latency convolution operation method, wherein the filter matrix includes filter blocks whose channel dimensions are listed in the horizontal and vertical directions and whose width dimension is listed in the depth direction.
5. In paragraph 4, Before step (a) above A low-latency convolution operation method, wherein the matrix operation unit further includes a step of mapping a convolution operation dimension for a target calculation to a GEMM operation dimension and tiling the input matrix and the filter matrix into a plurality of blocks of a predetermined size.
6. In paragraph 4, The above matrix operation unit A low-latency convolution operation method for producing the output block based on the horizontal position of the input block and the depth position of the filter block, according to Equation 1. <Formula 1> Wo+Wf-1 Here, Wo is the output width dimension, and Wf is the filter width dimension.
Citation Information
Patent Citations
Non-pneumatic tire with reduced tread deformation
KR1020230023294A
Automatic welding system
KR102211230B1
Protection element and battery pack including that
KR102381932B1
Antimicrobial sheet for mask
KR102955513B1
Programmable coarse grained and sparse matrix compute hardware with advanced scheduling
US20220164916A1
