Artificial neural network-based convolution operation method and apparatus using data flow optimization method

The method optimizes convolution operations on sparse data by varying buffer sizes and adjusting tiling sizes within the artificial neural network, addressing inefficiencies in existing technologies and enhancing the speed of object recognition and autonomous driving applications.

WO2024237652A1PCT designated stage expired Publication Date: 2024-11-21INDUSTRY UNIVERSITY COOPERATION FOUNDATION HANYANG UNIVERSITY
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/006514
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2024-05-13
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Conventional convolution operations on sparse data in artificial neural networks face inefficiencies due to irregular data access patterns and buffer utilization, particularly in object recognition tasks for autonomous driving, where sparse point cloud data requires irregular memory access and lacks optimal data flow optimization for sparse convolution operations.

Method used

An artificial neural network-based convolution calculation method that optimizes data flow by varying buffer sizes based on sparse input data characteristics, classifying kernel weights into groups, and adjusting tiling sizes to maximize data reuse and buffer efficiency, enabling faster convolution operations through improved data flow management.

Benefits of technology

This approach significantly enhances the speed of convolution operations, improves object recognition in 3D space, and supports high-dimensional autonomous driving and RGB-D-based robot navigation by maximizing data reuse and reducing computational overhead.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024006514_21112024_PF_FP_ABST
    Figure KR2024006514_21112024_PF_FP_ABST
Patent Text Reader

Abstract

An artificial neural network-based convolution operation method using a data flow optimization method according to an embodiment may include: an index data extraction step of extracting input index data for sparse input data included in input data; a kernel weight group generation step of classifying weights of a kernel according to a preset criterion on the basis of the size of the kernel and the input index data and then generating a kernel weight group on the basis of classified information; a data flow rule generation step of generating a data flow rule by mapping corresponding input index data to the kernel weight group; and a convolution operation step of performing a convolution operation on the basis of the data flow rule.
Need to check novelty before this filing date? Find Prior Art

Description

Convolution operation method and device based on artificial neural network using data flow optimization method

[0001] The present invention relates to a method and device for performing a convolution operation based on an artificial neural network using a data flow optimization method, and more specifically, to a technology for performing a convolution operation more quickly and effectively by optimizing a data flow so that data can be reused when performing convolution and operation using an artificial neural network.

[0002] Artificial Intelligence (AI) technology refers to a technology that realizes human learning abilities, reasoning abilities, perception abilities, and natural language understanding abilities through computer programs. Unlike conventional rule-based smart systems, it refers to a system in which machines learn, judge, and become smarter on their own.

[0003] Artificial intelligence technology consists of machine learning (deep learning) and its component technologies. Machine learning is an algorithmic technology that autonomously classifies and learns the characteristics of input data. Elementary technologies utilize machine learning algorithms like deep learning to mimic the cognitive and judgmental functions of the human brain. These technologies encompass areas such as linguistic understanding, visual understanding, inference / prediction, knowledge representation, and motion control.

[0004] With the advancement of artificial intelligence technology, it is being applied to autonomous driving, including recognizing objects around a vehicle. Specifically, object recognition methods based on Lidar / RGB-D sensors are the main method. Lidar / RGB-D sensor-based object recognition methods use point cloud data to identify the location and type of objects around the vehicle. The point cloud data is repeatedly subjected to convolution operations across multiple layers to extract features that can classify the objects in the data.

[0005] However, due to the nature of object recognition methods, point cloud data is sparsely distributed across space, and thus convolution operations are performed on these sparsely distributed data. Consequently, features extracted from each layer are stored irregularly in memory, necessitating irregular access to retrieve the feature data used in convolution operations. Consequently, performing convolution operations according to conventional techniques significantly increases the time required to complete the entire process.

[0006] Additionally, in convolution operations, the process of tiling (spatiotemporally dividing the operation) to create an optimal data flow (mapping) to fit the characteristics of the given hardware and buffer size is a very important part.

[0007] Conventional techniques target only the convolution of dense data with regular input-kernel-output sizes and operations depending on the tile size, making it difficult to optimize for sparse convolution of sparse data with irregular input-kernel-output sizes and operations.

[0008] Specifically, the convolution operation method based on sparse data performed through the process of Gather-GEMM-Scatter according to the prior art gathers only non-zero input data in a buffer, performs an operation with the corresponding kernel, and scatters the output data in the memory. Therefore, there is a problem that the number of reuses by index and the number of output data due to the blurring effect are irregular depending on the arrangement of the input data. Therefore, there is a disadvantage that it is impossible to evaluate the usage rate of the input / output buffer according to the tiling and the data reuse in each buffer due to the irregularity in the size of the input-kernel-output data and the number of operations.

[0009] A convolution operation method and device based on an artificial neural network using a data flow optimization method according to one embodiment is an invention designed to explain the above-described problem, and its purpose is to provide a method and device capable of efficiently performing a convolution operation based on the characteristics of sparse input data.

[0010] More specifically, the purpose is to effectively increase the overall speed of the convolution operation by minimizing the collection of unnecessary data by appropriately varying the size of the buffer using the location and size information of sparse input data when performing the convolution operation using an artificial neural network.

[0011] A convolution operation method based on an artificial neural network using a data flow optimization method according to one embodiment may include an index data extraction step of extracting input index data for sparse input data included in input data, a kernel weight group generation step of classifying kernel weights based on a size of a kernel and the input index data according to preset criteria, and then generating a kernel weight group based on the classified information, a data flow rule generation step of generating a data flow rule by mapping input index data corresponding to the kernel weight group, and a convolution operation step of performing a convolution operation based on the data flow rule.

[0012] The above kernel includes a 3X3 kernel, and the convolution operation may include a standard convolution or a stride convolution operation.

[0013] The above kernel weight group generation step may include a step of symmetrically generating a plurality of weight groups, taking into account the characteristics of a convolution operation based on a 3X3 kernel.

[0014] The above kernel weight group creation step may include a step of creating a 0th weight index (W0), a 2nd weight index (W2), a 6th weight index (W6), and an 8th weight index (W8) as a 0th weight group.

[0015] The above kernel weight group creation step may include a step of creating a first weight index (W1) and a seventh weight index (W7) as a first weight group.

[0016] The above kernel weight group creation step may include a step of creating a third weight index (W3) and a fifth weight index (W5) as a second weight group.

[0017] The above kernel weight group creation step may include a step of creating a fourth weight index (W4) as a third weight group.

[0018] The above data flow rule generation step may include a step of collecting input indices corresponding to the weight group and then sequentially storing the collected input indices in the input buffer for each weight group.

[0019] The above convolution operation includes deconvolution, and the kernel weight group generation step can determine the size of the weight group based on the sizes of the input buffer and the output buffer.

[0020] The above data flow rule generation step may include a tiling size adjustment step for adjusting a tiling size for the input data based on a ratio of the input index and an output index corresponding to the input index.

[0021] The above tiling size adjustment step may include a step of increasing the tiling size for an input channel in the case of standard convolution or stride convolution, and a step of decreasing the tiling size for an output channel in the case of deconvolution.

[0022] An artificial neural network-based convolution operation device using a data flow optimization method according to one embodiment may include an index data collection module that collects input index data for sparse input data included in input data, a data flow rule generation module that classifies kernel weights based on a kernel size and the input index data according to preset criteria, and then creates a kernel weight group based on the classified information and maps input index data corresponding to the kernel weight group to create a data flow rule, and a convolution operation module that performs a convolution operation based on the data flow rule.

[0023] The above kernel includes a 3X3 kernel, and the convolution operation may include a standard convolution or a stride convolution operation.

[0024] The above data flow rule generation module can symmetrically generate multiple weight groups by considering the characteristics of a convolution operation based on a 3X3 kernel.

[0025] The above data flow rule generation module can collect input indices corresponding to the weight group and then sequentially store the collected input indices in the input buffer for each weight group.

[0026] The above convolution operation includes deconvolution, and the data flow rule generation module can determine the size of the weight group based on the sizes of the input buffer and the output buffer.

[0027] According to one embodiment, a convolution operation device based on an artificial neural network using a data flow optimization method includes an index data collection module that collects input index data for sparse input data included in input data, a data flow rule generation module that classifies kernel weights based on a kernel size and the input index data according to preset criteria, and then creates a kernel weight group based on the classified information and maps input index data corresponding to the kernel weight group to create a data flow rule, and a convolution operation module that performs a convolution operation based on the data flow rule, wherein the data flow rule generation module can increase a tiling size for an input channel in the case of a standard convolution or a stride convolution, and can decrease a tiling size for an output channel in the case of a deconvolution.

[0028] A convolution operation method and device based on location information of sparse input data according to one embodiment collect and reuse data by considering the characteristics of input data and output data when performing a convolution operation based on sparse data, and therefore has the advantage of performing a convolution operation faster than conventional technologies.

[0029] In addition, in the case of convolution operations, there are many overlapping calculation processes due to their nature, but the convolution operation method according to the present invention has the advantage of being able to rapidly increase the operation speed of the convolution operation itself because it reuses information of input data collected in the past in the next operation.

[0030] In addition, these features can increase the speed of object recognition in three-dimensional space, so that high-speed forward obstacle recognition essential for high-dimensional autonomous driving can be efficiently performed, and RGB-D-based position estimation for fast and accurate robot navigation can also be efficiently performed.

[0031] The effects of the present invention are not limited to the technical problems mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0032] In order to more fully understand the drawings cited in the detailed description of the present invention, a brief description of each drawing is provided.

[0033] FIG. 1 is a diagram illustrating an example of input data input to a convolution operation device and an example of a kernel that performs a convolution operation according to one embodiment of the present invention.

[0034] Figure 2 is a diagram for explaining a process of performing a convolution operation based on sparse data according to a conventional technology.

[0035] Figure 3 is a diagram for explaining a problem that occurs when performing a convolution operation based on sparse data according to conventional technology.

[0036] FIG. 4 is a block diagram illustrating some components of a convolution operation device based on sparse data using an artificial neural network according to one embodiment of the present invention.

[0037] FIG. 5 is a diagram illustrating a method for adjusting the size of a tile by a data flow optimization module according to one embodiment of the present invention.

[0038] FIG. 6 and FIG. 7 are diagrams for explaining a method for a data flow optimization module according to one embodiment of the present invention to collect input data based on weight groups.

[0039] FIG. 8 and FIG. 9 are drawings for comparing and explaining a convolution operation method according to the present invention and a convolution operation method according to the prior art.

[0040] Figure 10 is a diagram comparing experimental results of a convolution operation method according to the present invention and a convolution operation method according to the prior art.

[0041] FIG. 11 is a diagram illustrating a method for a data flow optimization module according to one embodiment of the present invention to perform a scatter operation based on weight groups.

[0042] Figures 12 and 13 are drawings comparing experimental results of convolution operation speed according to the present invention and convolution operation speed according to the prior art.

[0043] The embodiments described in this specification and the configurations illustrated in the drawings are preferred examples of the disclosed invention, and there may be various modified examples that can replace the embodiments and drawings of this specification at the time of filing of this application.

[0044] Additionally, the terminology used herein is for the purpose of describing embodiments and is not intended to limit and / or restrict the disclosed invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0045] In this specification, terms such as “include,” “have,” or “have” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0046] Additionally, terms including ordinal numbers, such as “first,” “second,” etc., used herein may be used to describe various components, but the components are not limited by the terms.

[0047] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily practice them. Furthermore, in order to clearly explain the present invention, portions irrelevant to the description are omitted in the drawings.

[0048] Meanwhile, in this specification, the title of the invention is described as 'Artificial neural network-based convolution operation method and device using data flow optimization method', but for the convenience of explanation below, 'Artificial neural network-based convolution operation method using data flow optimization method' is abbreviated as 'convolution operation method', and 'Artificial neural network-based convolution operation device using data flow optimization method' is abbreviated as 'convolution operation device'.

[0049]

[0050] FIG. 1 is a diagram illustrating an example of input data input to a convolution operation device and an example of a kernel that performs a convolution operation according to one embodiment of the present invention.

[0051] The convolution operation begins with the process of analyzing input data (10) and collecting input index information. In the present invention, the (index) information refers to location information about a point where valid data, where actual data exists, is located within the input data.

[0052] For example, if data in a 5 X 5 matrix as shown in FIG. 1 is input as input data (the hatched part in the matrix represents an area where data exists, the unshaded part represents an area where data does not exist, and the data at the location where data exists is referred to as valid data), the index information extraction module (210) can extract information on the location where valid data is located within the input data as input index information.

[0053] Any known method for representing matrix data can be used to represent index information. A representative example is the Compressed Spare Row (CSR) format. For convenience of explanation, the index information in the present invention will be described based on the CSR format.

[0054] Specifically, the CSR format includes CSR_row information, which is information about how many valid data exist in each row sequentially, and CSR_col information, which is information about which column the valid data is located in each row sequentially. The CSR format coordinate representation method is generally the same as the matrix representation method, but there is a difference in that it starts from 0 instead of 1.

[0055] Therefore, the leftmost and topmost coordinates in the data are not expressed as (1,1), but as (0,0). Therefore, the input data can be viewed as having rows and columns starting from row 0 and column 0, and if the kernel is also a 3X3 matrix kernel, the offset of the kernel is (0,1,2) for rows and (0,1,2) for columns. In other words, both the input data and the kernel are expressed based on row 0.

[0056] If we sequentially explain the process of generating index information for the input data shown in Figure 1, the CSR_row value starts with 0 regardless of the data, and from the second value, it represents the accumulated value corresponding to the number of valid data in each row. Therefore, as shown in the figure, CSR_row starts with 0.

[0057] Since there is 1 valid data (I0) in row 0, the value of CSR_row considering up to row 0 is expressed as [0,1].

[0058] Since there is 1 valid data (I1) in the first row, the CSR_row value considering up to the first row is expressed as [0,1,2].

[0059] Since there are two valid data (I2, I3) in the second row, the CSR_row value considering up to the second row is expressed as [0,1,2,4].

[0060] Since there is one valid data (I4) in the third row, the CSR_row value considering up to the third row is expressed as [0,1,2,4,5].

[0061] Since there is one valid data (I5) in the 4th row, the CSR_row value considering up to the 4th row is expressed as [0,1,2,4,5,6].

[0062] CSR_col is information that sequentially informs in which column in each row the valid data is located. Therefore, based on Figure 1, in row 0, data exists in the 4th column, so CSR_col becomes [4], and in row 1, valid data exists in column 1, so CSR_col becomes [4,1]. In row 2, valid data exists in the 2nd and 3rd columns, so CSR_col becomes [4,1,2,3], and in row 3, valid data is located in column 3, so CSR_col becomes [4,1,2,3.3]. If data is generated in this manner, the final CSR_col becomes [4,1,2,3,3,2].

[0063]

[0064] FIG. 2 is a drawing for explaining a process of performing a convolution operation based on sparse data according to a conventional technology, and FIG. 3 is a drawing for explaining a problem that occurs when performing a convolution operation based on sparse data according to a conventional technology.

[0065] Referring to Fig. 2, the point cloud (20) of an object existing in a three-dimensional space obtained through LiDAR has sparse information as illustrated in the drawing. Therefore, rather than performing a convolution operation directly on these data, the input point cloud data undergoes a process of converting into a sparse pseudo image. (S10) In other words, the S10 process generates two-dimensional sparse convolution input data.

[0066] However, performing convolution operations on sparse image data can result in computational inefficiency. Therefore, sparse images are converted into dense data, and then dense convolution operations are performed based on the converted data. However, even when performing convolution operations through this process, the sparsity of sparse data generally exceeds 95%, so the substitution operation for dense convolution operations is highly inefficient.

[0067] Specifically, when explaining this in relation to a spatial architecture accelerator, a spatial architecture accelerator can efficiently process tensor operations, which account for most of neural network operations, and is therefore commonly used for neural network acceleration.

[0068] Tensor operations on an accelerator can be divided into data flows, which represent the partitioning (spatial division) and scheduling (temporal division) of data required to perform the operations.

[0069] Spatial architecture accelerators can have diverse data flows. Depending on how these data flows are configured, data reuse and hardware utilization within the accelerator can vary, significantly impacting performance and energy efficiency. Therefore, methods have been proposed to optimize performance and energy efficiency for a given accelerator configuration and tensor operations. This is called data flow optimization.

[0070] Existing data flow optimization methods only consider regular tensor operations applied to convolution operations based on dense data, which limits their optimization of irregular tensor operations applied to sparse convolution operations based on sparse data.

[0071] Specifically, convolution operations based on dense data perform regular tensor operations based on the characteristics of the kernel (height*width, stride, dilation), so they have the characteristics that the input data of a specific size (height*width), the size of the output data according to the kernel operation (height*width), data reuse in the memory layer, and the number of operations per index are determined by the kernel.

[0072] Conventional data flow optimization methods utilize these regularities to perform optimization, specifically, they divide operations into tile units for multiple dimensions of data to fit the input and output buffer sizes of the accelerator, and evaluate the performance of the data flow according to the tile processing order to determine the optimal tile size and processing order.

[0073] However, these methods have many difficulties in optimizing the irregular operation of sparse convolution of sparse data performed by the process of Gather-GEMM-Scatter.

[0074] Specifically, the convolution operation method based on sparse data performed through the process of Gather-GEMM-Scatter according to the prior art gathers only non-zero input data in a buffer, performs an operation with the corresponding kernel, and scatters the output data in the memory. Therefore, there is a problem that the number of reuses by index and the number of output data due to the blurring effect are irregular depending on the arrangement of the input data. Therefore, there is a disadvantage that it is impossible to evaluate the usage rate of the input / output buffer according to the tiling and the data reuse in each buffer due to the irregularity in the size of the input-kernel-output data and the number of operations.

[0075] That is, in the case of convolution operations based on sparse input data, problems arise in which the ratio of input-output indices and data reuse are irregular due to the placement of non-zero data in the input feature map and the blurring effect according to sparsity.

[0076] Specifically, if we look at this through Fig. 3, as shown in the upper part of Fig. 3, when a convolution operation is performed by applying a 3x3 kernel to the first tile (Tile1) area based on the same input index size (Ta=3) (based on striding 2), the ratio of the input index to the output index becomes 25 / 3, or 8.33, but when a convolution operation is performed by applying a 3x3 kernel based on the second tile (Tile2) area (based on striding 2), the ratio of the input index to the output index becomes 15 / 3, or 5.

[0077] That is, due to the irregular ratio of input-output indices, it is difficult to predict the usage of input and output buffers, and due to the irregular reuse of kernel indices, there is a disadvantage in that it is difficult to predict the memory access of mapping. This phenomenon occurs more frequently in convolution operations based on sparse data than in convolution operations based on dense data, as illustrated in the lower graph of Figure 3.

[0078] Accordingly, a method and device for optimizing data flow of a convolution operation according to an embodiment of the present invention are inventions designed to solve the above-described problems, and the purpose of the present invention is to provide a data flow optimization method that takes into account the influence of the blurring effect according to the characteristics of the convolution operation (Conv, Strided Conv, Deconv) and the sparsity / size of the input feature map when performing the convolution operation.

[0079] More specifically, in the convolution operation for sparse input data performed through the process of Gather-GEMM-Scatter, the Gather-Scatter is performed based on the Weight Group to maximize the reuse of data, and the input channel size and output channel tile size are optimized according to the operation characteristics and sparsity of the layer to improve the imbalance in the usage of the input buffer and output buffer of the convolution operation under a fixed buffer size, thereby increasing the overall speed of the convolution operation. Let's take a closer look at the drawing below.

[0080]

[0081] FIG. 4 is a block diagram illustrating some components of a convolution operation device based on sparse data using an artificial neural network according to one embodiment of the present invention.

[0082] Referring to FIG. 4, a convolution operation device based on location information of sparse input data according to one embodiment may include a processor (200) and a memory module (300). The processor (200) may include an input data collection module (210), a data flow rule generation module (220), and a convolution operation module (230), and the memory module (300) may include an input memory (310), an input buffer (320), an output buffer (330), and an output memory (340).

[0083] Meanwhile, in the present invention, the kernel moved to perform the convolution operation is described based on a 3X3 kernel and a stride of 2. However, this is only one embodiment of the present invention, and the principles of the present invention can be applied to kernels having sizes such as 2X2, 4X4, 5X5, etc., and the principles of the present invention can be applied as is to a 3D kernel rather than a 2D kernel.

[0084] In addition, in FIG. 4, the processor (200) and the memory module (300) are described separately for convenience of explanation, and the input data collection module (210), the data flow rule generation module (220), and the convolution operation module (230) within the processor (200) are described as separate components, but this is for convenience of explanation, and the roles of each module may be performed by one processor (200).

[0085] The input data collection module (210) analyzes the input data (10) in the input memory (310), collects input index information of sparse input data within the input data (10), and collects sparse input data within the input data (10) according to the size of the input buffer (310).

[0086] The data flow rule generation module (220) generates rules for performing convolution operations by adjusting the tile size for input / output channels based on collected sparse input data and input index information, or by adjusting the input gather and output scatter based on weight groups. A detailed description thereof will be provided later.

[0087] The convolution operation module (230) performs a convolution operation based on the sparse input data collected by the input data collection module (210) and the rules generated by the data flow rule generation module (220), and transmits the output data generated by the convolution operation to the output buffer (330).

[0088] The memory module (300) is a module in which input data and output data are stored. Depending on its characteristics, the output memory (340) may be implemented as DRAM, and the input buffer (310), the weight buffer (320), and the output buffer (330) may be implemented as SRAM.

[0089] The input buffer (310) may temporarily store input data (10) containing sparse input data to be performed on the convolution operation.

[0090] The input buffer (310) can temporarily store sparse input data collected by the input data collection module (210).

[0091] The output buffer (330) can temporarily store output data performed by the convolution operation module (230), and the output memory (340) can store final output data.

[0092]

[0093] FIG. 5 is a drawing for explaining a method for a data flow optimization module according to one embodiment of the present invention to adjust the size of a tile, and specifically, is a drawing for explaining a data flow rule generated by a data flow rule generation module (220) when a convolution operation module (230) performs a standard convolution.

[0094] Referring to (a) of Fig. 5, when the sparsity of sparse input data in the input data (10) is high or the size of the input feature map is relatively large, a blurring effect occurs significantly when performing a convolution operation, resulting in an imbalance between the sizes of the output buffer and the input buffer.

[0095] Specifically, as illustrated in (a) of Fig. 5, the ratio of the output index to the input index becomes 36 / 6=6, which results in a fairly high imbalance. Therefore, in this case, the input index is used less, but the output index is used more, so the input buffer is used relatively less, but the output buffer is used relatively more, resulting in an imbalance in buffer usage.

[0096] However, as shown in (b) of Fig. 5, when the sparsity of sparse input data within the input data (10) is low or the size of the input feature map is relatively small, the blurring effect occurs relatively small when performing the convolution operation, resulting in a relatively small imbalance between the output buffer and input buffer sizes. Specifically, as shown in (b) of Fig. 5, the ratio of the output index to the input index becomes 75 / 36=2.08, so the imbalance is smaller than in the case of (a) of Fig. 5.

[0097] Accordingly, the data flow rule generation module (220) according to the present invention calculates the ratio of the input index and the output index, and if the calculated ratio is higher than a preset standard, increases the size of the tile for the input channel, thereby resolving the problem of size imbalance between the input buffer and the output buffer in a way in which the input index is used a lot. For example, the preset standard can be set to various numbers, but if it is 3 or more, the size of the tile can be increased, and as the ratio of the calculated input index and the output index increases, the size of the tile can also be set to increase proportionally.

[0098]

[0099] FIG. 6 and FIG. 7 are drawings for explaining a method for a data flow optimization module according to an embodiment of the present invention to collect input data based on weight groups, and are drawings for explaining a data flow rule generated by a data flow rule generation module (220) when a convolution operation module (230) performs strid convolution.

[0100] Fig. 6 (a) is a diagram illustrating a process in which an input index and a kernel weight index are convolutionally operated to generate output data, and Fig. 6 (b) is a diagram in which input indices corresponding to each kernel weight group are grouped in generating output data.

[0101] Referring to (a) of Fig. 6, when a convolution operation is performed while a 3x3 kernel slides on the input data (10), the 0th input index (O0), the 1st input index (O1), and the 5th input index (O5) are generated by the product of the input index and the weight index of the kernel corresponding thereto, as shown in the drawing.

[0102] In general, strided convolution has the characteristic of reusing data by grouping weight indices for the input indices. When a convolution operation is performed by a 3X3 kernel, the range of corresponding input indices can be grouped according to the position where the kernel weights are arranged, as shown in the drawing.

[0103] In the basic data flow, Ta input indices corresponding to all indices of the kernel's weights (W0 to W8) are gathered into the input buffer. In this case, among the Ta input indices in the input buffer, the input indices multiplied by a specific weight index are reduced by 1 / 4. Therefore, there is a disadvantage in that data reuse is reduced in the input buffer, making the convolution operation inefficient.

[0104] However, in the case of the present invention, in order to improve this, input data is collected based on weight groups by considering the characteristics of the kernel and the characteristics of the convolution operation, thereby maximizing the reuse of data.

[0105] Specifically, by collecting only the input indices corresponding to a specific weight group into the input buffer, the efficiency of the convolution operation can be greatly improved because all input indices collected for all weights of the specific weight group can be reused.

[0106] Looking at this in detail through the drawing, in the case of the 0th weight group (WG0), if only the input indices corresponding to the 0th, 2nd, 6th, and 8th weight indices included in the 0th weight group are collected in the input buffer, all input indices are always reused for all weight indices of the 0th weight group (WG0), so the efficiency of the operation can be improved.

[0107] Looking at this specifically by weight group, the 0th weight index (W0), the 2nd weight index (W2), the 6th weight index (W6), and the 8th weight index (W8) corresponding to the 0th weight group (WG0) perform convolution operations only with the 0th input index (I0), the 2nd input index (I2), the 4th input index (I4), the 6th input index (I6), the 17th input index (I17), the 19th input index (I19), the 21st input index (I21), the 23rd input index (I23), the 25th input index (I25), the 35th input index (I35), the 37th input index (I37), and the 39th input index (I39).

[0108] In addition, the first weight index (W1) and the seventh weight index (W7) corresponding to the first weight group (WG1) perform convolution operations only with the first input index (I1), the third input index (I3), the fifth input index (I5), the eighteenth input index (I18), the twentieth input index (I20), the twenty-second input index (I22), the twenty-fourth input index (I24), the twenty-sixth input index (I26), the thirty-fourth input index (I34), the thirty-sixth input index (I36), and the thirty-eighth input index (I38).

[0109] In addition, the third weight index (W3) and the fifth weight index (W5) corresponding to the second weight group (WG2) perform convolution operations only with the seventh input index (I7), the ninth input index (I9), the eleventh input index (I11), the thirteenth input index (I13), the fifteenth input index (I15), the twenty-eighth input index (I28), the thirtieth input index (I30), and the thirty-second input index (I32).

[0110] In addition, the fourth weight index (W4) corresponding to the third weight group (WG3) performs a convolution operation only with the eighth input index (I8), the tenth input index (I10), the twelfth input index (I12), the fourteenth input index (I14), the sixteenth input index (I16), the twenty-seventh input index (I27), the twenty-ninth input index (I29), the thirty-first input index (I31), and the thirty-third input index (I33).

[0111] By collecting input data in this way and performing convolution operations, all weight indices have the opportunity for data reuse (Reduction) for the output indices. Therefore, as illustrated in FIG. 7, after completing the Gather-GEMM operation for all weight groups for the input indices collected in the input buffer (310), the output indices for which reduction has been completed can be scattered from the output buffer (330) to the output memory (340), thereby achieving 100% data reuse for input-output.

[0112]

[0113] FIG. 8 and FIG. 9 are drawings for comparing and explaining a convolution operation method according to the present invention and a convolution operation method according to the prior art. Specifically, FIG. 8 is a drawing for explaining an operation process when a weight group-based convolution operation method according to the prior art is not applied, and FIG. 9 is a drawing for explaining an operation process when a weight group-based convolution operation method according to the present invention is applied.

[0114] Referring to Fig. 8, when performing a convolution operation according to the prior art, the collection of input indices is divided into two parts for the entire input data due to the limited size of the input buffer (first collection (Ta=27), second collection (Ta=13)).

[0115] And when performing a convolution operation based on such collected data, as shown in the drawing, the number of indices (Ta,eff) actually used for the operation in the array decreases compared to the number of input indices (Ta) collected in the input buffer, so the output buffer cannot efficiently reuse the input indices.

[0116] That is, since the capacity of the output buffer cannot be used efficiently, the number of times the input index is collected increases, which increases the number of times the weight data is accessed to the output memory (DRAM), resulting in a problem in which the convolution operation speed is reduced.

[0117] However, when performing a convolution operation according to the present invention, input indices are collected from input data in units of weight groups as illustrated in FIG. 9, so that data can be collected for a wider index range, which can reduce the number of times input indices are collected.

[0118] In addition, since the number of indices (Ta,eff) actually used for operations in the array is almost the same as the number of input indices collected in the input buffer (Ta), the output buffer can reuse most of the input indices, which has the advantage of increasing the overall speed of the convolution operation.

[0119] FIG. 10 is a diagram comparing experimental results of a convolution operation method according to the present invention and a convolution operation method according to a conventional technique. FIG. 10 (1) is a graph showing experimental results according to a convolution operation according to a conventional technique, FIG. 10 (2) is a graph showing experimental results according to a weight group-based convolution operation according to the present invention, and FIG. 10 (3) is a diagram showing experimental results according to a convolution operation applying a weight group-based and tiling optimization method according to the present invention.

[0120] Referring to (1) of Fig. 10, when performing a convolution operation according to the prior art, the reuse ratio of input data is only about 1 / 4 (255 / 1023), but when performing a convolution operation based on a weight group according to the present invention, it can be seen that the reuse ratio of input data is 100% (358 / 358), as shown in (2) of Fig. 10.

[0121] In addition, when performing a convolution operation by further applying the tiling technique according to the present invention, as shown in (3) of Fig. 10, not only is the reuse ratio of input data 100% (716 / 716), but the collection range of input indices is also expanded, so there is an advantage of performing the operation more quickly.

[0122]

[0123] FIG. 11 is a diagram for explaining a method in which a data flow optimization module according to one embodiment of the present invention performs a scatter operation based on a weight group, and is a diagram for explaining a data flow rule generated by a data flow rule generation module (220) when a convolution operation module (230) performs deconvolution.

[0124] In the case of deconvolution, unlike the strided convolution described above, when performing a convolution operation, all weight indices share the input index, but due to its nature, it is difficult to reuse the output index between weight indices. Therefore, in the case of deconvolution, input data is first temporarily collected and stored in the input buffer (210), and then the weights are collected by grouping them according to a certain standard as shown in the drawing, and when the operation on the weight group is finished, a scatter operation is performed to create a data flow rule, thereby maximizing the reuse of the input data.

[0125] The size of the weight group can be adjusted in a pattern of 16x1, 8x2, 4x4, 2x8, and 1x16 depending on the size of the output buffer (330), thereby minimizing the effect of the size imbalance between the input buffer and the output buffer.

[0126]

[0127] Meanwhile, an artificial neural network-based convolution operation device using a data flow optimization method according to the present invention can perform tiling optimization for input channels and output channels described in FIG. 5 according to the usage of input buffers and output buffers after setting the data flow to maximize data reuse in the operation.

[0128] In the case of convolution operations, the blurring effect varies depending on the location and distribution of sparse input data within the input data, and thus the ratio of the input and output indices is determined accordingly. For example, the ratio is set to 1.5x or more for general convolution, 0.5x for strided convolution, and 1x, 4x, or 16x for deconvolution, depending on the weight size.

[0129] When performing a convolution operation, if the usage of a specific buffer becomes 100%, the speed of the convolution operation is hindered. Therefore, when the usage of the output buffer becomes 100% (i.e., the usage of the input buffer is lower than that of the output buffer), the tile size for the input channel is increased to increase the usage of the input buffer, thereby preventing the problem of the speed of the operation being hindered due to the usage of the buffer.

[0130] Conversely, when the input buffer usage reaches 100% (i.e., the output buffer usage falls short of the input buffer), the problem of the operation speed being reduced due to buffer usage can be prevented by increasing the tile size for the output channel to increase the output buffer usage.

[0131]

[0132] Figures 12 and 13 are drawings comparing experimental results of convolution operation speed according to the present invention and convolution operation speed according to the prior art.

[0133] Referring to FIG. 12, when performing strided convolution according to the present invention, it can be seen that the reuse of input data can be improved from 50% to a maximum of 100% through the weight group-based input collection data flow. Through this, it can be seen that the computational overhead (Non-Execution Cycle compared to the total computation) due to the reduction in data reuse is reduced by 62.9% (12.4% -> 6%).

[0134] When performing deconvolution according to the present invention, it can be seen that the computational overhead is reduced by 56.9% (29.9% -> 2.9%) compared to the prior art because output data reuse and buffer usage can be optimized when executing data flow through weight group-based scatter operation.

[0135] Furthermore, when performing a convolution operation using a tiling technique according to the present invention, the imbalance in the usage of input and output buffers due to the blurring effect can be improved through tiling that takes into account the operation characteristics and sparsity of each layer in a convolution processing device with a fixed buffer size. It can be seen that this method reduces the operation overhead by 20% (4.5% -> 3.6%).

[0136] When comprehensively applying this and examining the overall speed of the convolution operation, it can be seen that the execution time of the convolution operation is reduced by 14.5% in the case of the convolution operation applying the data flow according to the conventional technology and the convolution operation applying the data flow according to the present invention, as shown in FIG. 13.

[0137] So far, we have looked at the configuration and process of the present invention in detail through drawings.

[0138] A convolution operation method and device based on location information of sparse input data according to one embodiment collect and reuse data by considering the characteristics of input data and output data when performing a convolution operation based on sparse data, and therefore has the advantage of performing a convolution operation faster than conventional technologies.

[0139] In addition, in the case of convolution operations, there are many overlapping calculation processes due to their nature, but the convolution operation method according to the present invention has the advantage of being able to rapidly increase the operation speed of the convolution operation itself because it reuses information of output data collected in the past in the next operation.

[0140] In addition, these features can increase the speed of object recognition in three-dimensional space, so that high-speed forward obstacle recognition essential for high-dimensional autonomous driving can be efficiently performed, and RGB-D-based position estimation for fast and accurate robot navigation can also be efficiently performed.

[0141] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used singly; however, those skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors, or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0142] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0143] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0144] Although the embodiments have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents. Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims

1. A method for performing a convolution operation using a processor, an input buffer, and an output buffer, An index data collection step for collecting input index data for sparse input data included in the input data; A kernel weight group generation step of generating a kernel weight group based on the classified information after classifying the weights of the kernel according to preset criteria based on the size of the kernel and the input index data; A data flow rule generation step for generating a data flow rule by mapping input index data corresponding to the above kernel weight group; and A convolution operation step for performing a convolution operation based on the above data flow rule; A convolution operation method based on an artificial neural network using a data flow optimization method.

2. In paragraph 1, The above kernel includes a 3X3 kernel, The above convolution operation includes a standard convolution or a stride convolution operation, The above kernel weight group creation step is: The above weight group includes a step of symmetrically generating multiple weight groups, taking into account the characteristics of the convolution operation based on the 3X3 kernel. A convolution operation method based on an artificial neural network using a data flow optimization method.

3. In paragraph 2, The above kernel weight group creation step is: A step of generating a 0th weight index (W0), a 2nd weight index (W2), a 6th weight index (W6), and an 8th weight index (W8) as a 0th weight group, A convolution operation method based on an artificial neural network using a data flow optimization method.

4. In paragraph 2, The above kernel weight group creation step is: Including a step of generating a first weight index (W1) and a seventh weight index (W7) as a first weight group, A convolution operation method based on an artificial neural network using a data flow optimization method.

5. In paragraph 2, The above kernel weight group creation step is: Including a step of generating a third weight index (W3) and a fifth weight index (W5) as a second weight group, A convolution operation method based on an artificial neural network using a data flow optimization method.

6. In paragraph 2, The above kernel weight group creation step is: Including a step of creating a fourth weight index (W4) as a third weight group, A convolution operation method based on an artificial neural network using a data flow optimization method.

7. In paragraph 2, The above data flow rule creation step is: A step of collecting input indices corresponding to the weight group, and then sequentially storing the collected input indices in the input buffer for each weight group; A convolution operation method based on an artificial neural network using a data flow optimization method.

8. In paragraph 1, The above convolution operation includes deconvolution, The above kernel weight group creation step is: Determine the size of the weight group based on the size of the input buffer and output buffer, A convolution operation method based on an artificial neural network using a data flow optimization method.

9. In paragraph 1, The above data flow rule creation step is: A tiling size adjustment step for adjusting a tiling size for the input data based on a ratio of the input index and an output index corresponding to the input index, A convolution operation method based on an artificial neural network using a data flow optimization method.

10. In paragraph 9, The above tiling size adjustment step is: In the case of standard convolution or stride convolution, a step of increasing the tiling size for the input channel is included, and in the case of deconvolution, a step of decreasing the tiling size for the output channel is included. A convolution operation method based on an artificial neural network using a data flow optimization method.

11. An index data collection module that collects input index data for sparse input data included in the input data; A data flow rule generation module that generates a data flow rule by classifying the weights of the kernel according to preset criteria based on the size of the kernel and the input index data, and then generating a kernel weight group based on the classified information and mapping the input index data corresponding to the kernel weight group; A convolution operation module that performs a convolution operation based on the above data flow rule; A convolutional operation device based on an artificial neural network using a data flow optimization method.

12. In paragraph 11, The above kernel includes a 3X3 kernel, The above convolution operation includes a standard convolution or a stride convolution operation, The above data flow rule generation module, The above weight group symmetrically generates multiple weight groups, considering the characteristics of the convolution operation based on the 3X3 kernel. A convolutional operation device based on an artificial neural network using a data flow optimization method.

13. In paragraph 12, The above data flow rule generation module, After collecting input indices corresponding to the above weight group, the collected input indices are sequentially stored in the input buffer for each weight group. A convolutional operation device based on an artificial neural network using a data flow optimization method.

14. In paragraph 11, The above convolution operation includes deconvolution, The above data flow rule generation module, Determine the size of the weight group based on the size of the input buffer and output buffer, A convolutional operation device based on an artificial neural network using a data flow optimization method.

15. Index data collection module that collects input index data for sparse input data included in the input data; A data flow rule generation module that generates a data flow rule by classifying the weights of the kernel according to preset criteria based on the size of the kernel and the input index data, and then generating a kernel weight group based on the classified information and mapping the input index data corresponding to the kernel weight group; A convolution operation module that performs a convolution operation based on the above data flow rule; The above data flow rule generation module, In the case of standard convolution or stride convolution, the tiling size for the input channel is increased, and in the case of deconvolution, the tiling size for the output channel is decreased. A convolutional operation device based on an artificial neural network using a data flow optimization method.

Citation Information

Patent Citations

  • Sola power generating apparatus

    KR1020220148489A

  • Cast-in-place type moisturizing porous concrete mixture for plant slopes or revetments using fiber, and its construction method

    KR102352110B1

  • Neural network device, Neural network processor and method of operating neural network processor

    KR102390379B1

  • System and method for executing convolution in a neural network

    US20200218978A1

  • Sparsity-aware neural processing unit for performing constant probability index matching and processing method of the same

    US20210303980A1