Method and apparatus for processing data

Abstract

A method for processing data, comprising: manipulating input data based on a configuration of the input data and a configuration of hardware for processing the input data to generate manipulated data; rearranging the manipulated data based on sparsity of the manipulated data to generate rearranged data; and processing the rearranged data to generate output data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0161677 filed in the Korean Intellectual Property Office on December 6, 2019, the disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0003] The following description relates to a method and apparatus for processing data. Background Art

[0004] Neural network refers to a computing architecture that uses the biological brain as a model. According to the development of neural network technology, input data is analyzed and effective information is extracted by using neural network devices in various types of electronic systems.

[0005] Neural network devices perform a large number of operations on input data. Research has been conducted on technologies that can efficiently process neural network operations. Summary of the invention

[0006] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0007] Various aspects provide a method and apparatus for processing data, and a computer-readable recording medium having recorded thereon a program for executing the method on a computer.

[0008] In one general aspect, a method for processing data includes: manipulating input data based on a configuration of the input data and a configuration of hardware for processing the input data to generate manipulated data; rearranging the manipulated data based on sparsity of the manipulated data to generate rearranged data; and processing the rearranged data to generate output data.

[0009] In another general aspect, a computer-readable recording medium includes: a method of executing the above method by using a computer.

[0010] In another general aspect, an apparatus comprises: a memory storing at least one program; and a processor configured to execute the at least one program to perform the following operations: manipulating input data based on a configuration of the input data and a configuration of hardware for processing the input data to generate manipulated data; rearranging the manipulated data based on sparsity of the manipulated data to generate rearranged data; and processing the rearranged data to generate output data.

[0011] Manipulation of input data can be applied: Hardware-based configuration adds at least one channel of zero configuration to the input data.

[0012] The input data may be manipulated based on the number of first channels included in the input data and the number of second channels included in the hardware.

[0013] The input data may be manipulated based on a value obtained by dividing the number of second channels by the number of first channels.

[0014] Manipulating the input data may include adding n channels between the first channels, each of the n channels being configured by zero, and n may be a natural number less than or equal to a value obtained by dividing the number of second channels by the number of first channels.

[0015] Manipulating the input data may include shifting elements of one or more columns included in the input data according to a specified rule.

[0016] The specified rule may include shifting elements of one or more columns by a specified size in the same direction, and the specified rule may be periodically applied to the one or more columns.

[0017] Rescaling the manipulated data may include shifting at least one element included in the manipulated data from a first position of a first column including the at least one element to a second position of a second column.

[0018] The first position of the first column and the second position of the second column may correspond to each other.

[0019] The first position of the first column and the second position of the second column may be different from each other.

[0020] In another general aspect, a device includes a memory and a processor configured to execute at least one program stored in the memory to perform the following operations: generate first data by manipulating input data based on the number of channels of an operator included in the processor; generate second data by rearranging the first data based on the validity of elements included in the first data; and perform a convolution operation on the second data to generate output data.

[0021] Generating the second data may include replacing at least one invalid element in the first data with a valid element.

[0022] At least one invalid element can be zero, while valid elements can be non-zero numbers.

[0023] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram showing the architecture of a neural network.

[0025] Figure 2 and Figure 3 is a diagram illustrating an example of a convolution operation of a neural network.

[0026] Figure 4 is a configuration diagram showing an example of an apparatus for processing data.

[0027] Figure 5 is a flowchart illustrating an example of a method of processing data.

[0028] Figure 6 and Figure 7 is a diagram showing an example of input data and operators.

[0029] Figure 8 is a diagram showing an example of a processor manipulating input data.

[0030] Fig.9A and Fig. 9B is a diagram showing an example of a processor rearranging manipulated data.

[0031] Fig.10 is a diagram showing another example of a processor rearranging manipulated data.

[0032] Fig.11 is a diagram showing another example of a processor rearranging manipulated data.

[0033] Fig.12 is a diagram showing another example of a processor rearranging manipulated data.

[0034] Fig.13 is a diagram showing another example of a processor rearranging manipulated data.

[0035] Fig.14 is a diagram showing another example of a processor rearranging manipulated data.

[0036] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals should be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0037] The following detailed description is provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to those order of operations set forth herein, but may be significantly changed after understanding the disclosure of the present application, except for operations that must be performed in a certain order. In addition, for greater clarity and brevity, the description of known features may be omitted after understanding the disclosure of the present application.

[0038] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices and / or systems described herein, which will be apparent after understanding the disclosure of the present application.

[0039] Throughout the specification, when a component is described as being "connected to" or "coupled to" another component, it may be directly "connected to" or "coupled to" the other component, or there may be one or more other components interposed therebetween. Conversely, when an element is described as being "directly connected to" or "directly coupled to" another element, there may be no other elements interposed therebetween. Likewise, similar expressions such as "between" and "directly between" and "adjacent to" and "immediately adjacent to" should also be understood in the same manner. As used herein, the term "and / or" includes any one of the items listed in the association and any combination of any two or more.

[0040] Although terms such as "first", "second", "third" and the like may be used herein to describe various components, assemblies, regions, layers or parts, these components, assemblies, regions, layers or parts should not be limited by these terms. Instead, these terms are only used to distinguish one component, component, region, layer or part from another component, component, region, layer or part. Therefore, without departing from the teachings of the examples described herein, the first component, component, region, layer or part mentioned in the examples may also be referred to as the second component, component, region, layer or part.

[0041] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the articles "a", "an" and "the" are also intended to include plural forms. The terms "include", "comprise" and "have" indicate the presence of the described features, numbers, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, numbers, operations, components, elements and / or combinations thereof.

[0042] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by a person of ordinary skill in the art to which the present disclosure belongs based on an understanding of the disclosure of the present application. Terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant technology and / or the present application, and should not be interpreted as ideal or overly formal meanings unless explicitly defined as such herein. In this article, the use of the term "may" with respect to an example or embodiment (e.g., with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment in which such a feature is included or implemented, and all examples are not limited thereto.

[0043] Hereinafter, examples will be described in detail with reference to the accompanying drawings.

[0044] Figure 1 is a schematic diagram showing the architecture of a neural network.

[0045] refer to Figure 1 , the neural network 1 may be a deep neural network (DNN) or an n-layer neural network architecture. The DNN or n-layer neural network may correspond to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzman machine, etc. For example, the neural network 1 may be implemented as a convolutional neural network (CNN), but is not limited thereto. Figure 1 Some convolutional layers in a convolutional neural network corresponding to the example of neural network 1 are shown, but in addition to the shown convolutional layers, the convolutional neural network also includes pooling layers, fully connected layers, etc.

[0046] The neural network 1 may be implemented by an architecture having multiple layers, including an input image, a feature map, and an output. In the neural network 1, a convolution operation using a filter (also referred to as a kernel) is performed on the input image, and as a result, a feature map is output. At this time, the generated output feature map is used as the input feature map, and a convolution operation using the kernel is performed again on it, and a new feature map is output. As a result of repeatedly performing this convolution operation, a recognition result of the characteristics of the input image by the neural network 1 can be finally output.

[0047] For example, when an image with a size of 24×24 pixels is input to Figure 1When the neural network 1 is used, the input image can be output as a 4-channel feature map with a size of 20×20 pixels by using a convolution operation of the kernel. Subsequently, the size of the 20×20 feature map is reduced by using an iterative convolution operation of the kernel, and a feature of a size of 1×1 pixels can be output. The neural network 1 can filter and output robust features that can represent the entire image from the input image by repeatedly performing convolution operations and subsampling (or pooling) operations at each layer, and the recognition result of the input image can be obtained by the final features output.

[0048] Figure 2 and Figure 3 is a diagram illustrating an example of a convolution operation of a neural network.

[0049] refer to Figure 2 , assuming that the input feature map 210 has a 6×6 pixel size, the kernel 220 has a 3×3 pixel size, and the output feature map 230 has a 4×4 pixel size, but the example is not limited thereto. The neural network can be implemented using feature maps and kernels of various sizes. In addition, the values ​​defined in the input feature map 210, the kernel 220, and the output feature map 230 are only exemplary values, and various examples are not limited thereto.

[0050] The kernel 220 performs a convolution operation while sliding in units of a region (or tile) of 3×3 pixels in the input feature map 210. The convolution operation is an operation in which a multiplication is performed between each pixel value of a certain region of the input feature map 210 and a weight as an element of the corresponding kernel 220, and the values ​​obtained by the multiplication are added to obtain each pixel value of the output feature map 230.

[0051] First, the kernel 220 performs a convolution operation on the first region 211 of the input feature map 210. That is, the pixel values ​​1, 2, 3, 4, 5, 6, 7, 8, and 9 of the first region 211 are multiplied by weights -1, -3, +4, +7, -2, -1, -5, +3, and +1, which are elements of the kernel 220, respectively, and the results -1, -6, 12, 28, -10, -6, -35, 24, and 9 are obtained. Next, 15 is obtained by adding the acquired values ​​1, -6, 12, 28, -10, -6, -35, 24, and 9, so the pixel value (231) of the first row and the first column of the output feature map 230 is determined to be 15. Here, the pixel value (231) of the first row and the first column of the output feature map 230 corresponds to the first region 211.

[0052] In the same manner as described above, by performing a convolution operation between the second region 212 of the input feature map 210 and the kernel 220, the pixel value (232) of the first row and the second column of the output feature map 230 is determined. Finally, by performing a convolution operation between the sixteenth region 213 (i.e., the last window of the input feature map 210) and the kernel 220, the pixel value (233) of the fourth row and the fourth column of the output feature map 230 is determined to be 11.

[0053] although Figure 2 A two-dimensional convolution operation is shown, but the convolution operation may correspond to a three-dimensional convolution operation where there are multiple channels of input feature maps, kernels, and output feature maps. Figure 3 Describe this.

[0054] refer to Figure 3 , the input feature map 201 may have a three-dimensional size, there may be X input channels, and the two-dimensional input feature map of each input channel may have a size of H rows and W columns (X, W, and H are natural numbers). The kernel 202 may have a four-dimensional size, and a two-dimensional kernel having a size of R rows and S columns may have up to X input channels and Y output channels (R, S, and Y are natural numbers). In other words, the kernel 202 may have a number of channels corresponding to the number X of input channels of the input feature map 201 and the number Y of output channels of the output feature map 203, and the two-dimensional kernel of each channel may have a size of R rows and S columns. The output feature map 203 may be generated by a three-dimensional convolution operation between the three-dimensional input feature map 201 and the four-dimensional kernel 202, and according to the three-dimensional convolution operation result, there may be Y channels.

[0055] The process of generating an output feature map through a convolution operation between a two-dimensional input feature map and a two-dimensional kernel is similar to the above reference. Figure 2 The same as described above, and by repeatedly performing a two-dimensional convolution operation (refer to Figure 2 ), between the input feature map 201 having X input channels and the kernel 202 having X input channels and Y output channels, an output feature map 203 having Y output channels can be generated.

[0056] Figure 4 is a configuration diagram showing an example of an apparatus for processing data.

[0057] refer to Figure 4 , the apparatus 400 for processing data includes a memory 410 and a processor 420. Figure 4 Although not shown in the figure, the apparatus 400 for processing data may be connected to an external memory. Figure 4 The device 400 includes only configuration elements related to this example. Figure 4In addition to the configuration elements shown in , other general configuration elements may also be included in the device 400.

[0058] The device 400 may be implemented as described above. Figures 1 to 3 An apparatus for the described neural network. For example, the apparatus 400 can be implemented using various devices such as a personal computer (PC), a server device, a mobile device, and an embedded device. As a detailed example, the apparatus 400 for processing data may be included in a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, a robot, a medical instrument, and the like, which perform speech recognition, image recognition, image classification, and the like by using a neural network, but are not limited thereto. In addition, the apparatus 400 may correspond to a dedicated hardware accelerator (HW accelerator) installed in the above-mentioned device, and may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), a neural engine, and the like, which are dedicated modules for driving a neural network.

[0059] The memory 410 stores various data processed by the device 400. For example, the memory 410 may store data processed by the device 400 and data to be processed by the device 400. In addition, the memory 410 may store applications to be driven by the device 400, driver programs, and the like.

[0060] For example, memory 410 may include random access memory (RAM) (e.g., dynamic random access memory (DRAM) or static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or optical disk storage, a hard disk drive (HDD), a solid-state drive (SSD), or flash memory.

[0061] The processor 420 controls the overall functions for driving the neural network in the device 400. For example, the processor 420 generally controls the device 400 by executing a program stored in the memory 410. The processor 420 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. provided in the device 400, but is not limited thereto.

[0062] The processor 420 reads and / or writes data (e.g., image data, feature map data, kernel data, etc.) from and to the memory 410, and implements a neural network by using the read or written data. When implementing the neural network, the processor 420 drives the processing unit included therein to repeatedly perform a convolution operation between the kernel and the input feature map to generate data related to the output feature map. At this time, the number of convolution operations may be determined depending on various factors (e.g., the number of channels of the input feature map, the number of channels of the kernel, the size of the input feature map, the size of the kernel, and the accuracy of the value).

[0063] For example, the processing unit may include a logic circuit for a convolution operation. The processing unit may include an operator implemented by a combination of a multiplier, an adder, and an accumulator. In addition, the multiplier may be implemented by a combination of multiple sub-multipliers, and the adder may also be implemented by a combination of multiple sub-adders.

[0064] The processor 420 may also include an on-chip memory that performs a cache function to process convolution operations, and a dispatcher for dispatching various operands, such as pixel values ​​of an input feature map or weight values ​​of a kernel. For example, the dispatcher may dispatch operands (such as pixel values ​​and weight values) required for an operation to be performed by a processing unit from data stored in the memory 410 to the on-chip memory. The dispatcher then dispatches the operands dispatched in the on-chip memory back to the processing unit for convolution operations.

[0065] The performance of the device 400 depends on the input data and the hardware configuration of the device 400. For example, when the number of channels of the operator included in the processor 420 is N (N is a natural number), and the number of channels of the input data (input feature map data and kernel data) is not a multiple of N, the performance of the device 400 may be reduced. When the number of channels of the input data is less than the number of channels of the operator, there are idle channels in the operator. Therefore, the device 400 may not be able to operate at the highest performance.

[0066] The device 400 identifies input data and the configuration of hardware for processing the input data. Then, the device 400 manipulates the input data based on the recognition result. Here, manipulating the input data refers to adding at least one channel to the input data. Therefore, the device 400 can process data when there is no idle channel in the operator.

[0067] In addition, the device 400 rearranges the manipulated data based on the sparsity of the manipulated data. Here, rearranging the manipulated data refers to a process of changing the initial configuration of the data matrix, such as changing the positions of some elements included in the data matrix, or skipping some rows or columns included in the data matrix. Therefore, the device 400 can output a valid result without performing unnecessary operations, thus reducing the total number of operations while outputting the desired result.

[0068] Reference Figures 5 to 14 An example in which the apparatus 400 manipulates input data, rearranges the manipulated data, and processes the rearranged data to generate output data is described.

[0069] Figure 5 is a flowchart illustrating an example of a method of processing data.

[0070] refer to Figure 5 , the method for processing data is configured by Figure 4 The apparatus 400 shown in FIG. 1 performs the processing steps in a time series. Therefore, it can be seen that although the following omits the Figure 4 400, but the above description also applies to Figure 5 The data processing method shown in .

[0071] In operation 510 , the processor 420 manipulates input data based on the input data and the configuration of hardware for processing the input data.

[0072] Input data refers to the target of the convolution operation to be performed by the processor 420. For example, the input data may include image data, feature map data, or kernel data. In this case, the feature map data may be input feature map data or output feature map data. The processor 420 performs convolution operations in multiple layers, and the output feature map data in the previous layer becomes the input feature map data in the next layer. Therefore, the input data of operation 510 may be input feature map data or output feature map data. As described above with reference to Figure 4 As described, the input data may be a matrix in which data is included as elements.

[0073] The processor 420 manipulates the input data based on the input data and the configuration of the hardware. Here, the configuration of the hardware refers to the number of channels of the operator included in the processor 420.

[0074] For example, processor 420 can identify the configuration of input data and operator, and compare the number of the channel of input data with the number of the channel of operator. Processor 420 can add at least one zero channel configured by zero to input data based on comparison result. At this time, the number of the channel added to input data is determined depending on the number of the channel of operator. For example, processor 420 can add at least one channel to input data so that there is no idle channel in operator.

[0075] In the following, reference will be made to Figure 6 and Figure 7 An example of the configuration of the processor 420 identifying input data and hardware is described below. Figure 8 An example of processor 420 manipulating input data is described below.

[0076] Figure 6 and Figure 7 Examples of input data and operators are shown.

[0077] Figure 6 and Figure 7 Input feature map data 610 and 710 and kernel data 620 and 720 are shown as examples of input data. Figure 6 and Figure 7 Examples of operators 630 and 730 that perform operations between input data are shown.

[0078] exist Figure 6 and Figure 7 , the operators 630 and 730 are shown to include 16 channels, but are not limited thereto. That is, the number of channels included in the operators 630 and 730 may vary depending on the specifications of the apparatus 400 for processing data.

[0079] refer to Figure 6 , the input feature map data 610 has a three-dimensional size, and there are 16 input channels therein. In addition, the two-dimensional data of each input channel has a size of two rows and three columns. In addition, the kernel data (weight) 620 has the number of channels corresponding to the number of input channels (16) of the input feature map data 610.

[0080] The input feature map data 610 and the kernel data 620 are input to the operator 630 and a convolution operation is performed. At this time, since the number of channels of the input feature map data 610 is the same as the number of channels of the operator 630, there are no idle channels in the operator 630. Therefore, the operator 630 can perform operations with the highest efficiency.

[0081] refer to Figure 7, there are three input channels in the input feature map data 710. In addition, the kernel data (weight) 720 has the number of channels corresponding to the number of input channels of the input feature map data 710 (3).

[0082] Since the operator 730 includes 16 channels, when the input feature map data 710 is input to the operator 730, 13 idle channels 731 are generated. Therefore, the operator 730 may not be able to perform operations at the highest efficiency.

[0083] In this case, the processor 420 may add channels to the input feature map data 710. For example, the processor 420 may set the number of channels of the input feature map data 710 to be the same as the number of channels of the operator 730 by adding 13 channels to the input feature map data 710.

[0084] In addition, the processor 420 can set the number of channels of the input feature map data 710 to M times the number of channels of the operator 730 by adding the channels to the input feature map data 710 (M is a natural number). Figure 7 As shown in , the processor 420 can set the number of channels of the input feature map data 710 to 16, 32, etc. by adding the channels to the input feature map data 710.

[0085] Therefore, the operator 730 can perform operations by using all channels, and thus can improve the operation efficiency of the operator 730. In other words, the processor 420 can manipulate the input feature map data 710 so that no idle channels are generated in the operator 730.

[0086] Figure 8 is a diagram showing an example of a processor manipulating input data.

[0087] Figure 8 Input data 810 and manipulated data 820 and 830 are shown. Here, the input data 810 may be input feature map data, but is not limited thereto. The processor 420 manipulates the input data 810 to generate first manipulated data 820, and manipulates the first manipulated data 820 to generate second manipulated data 830.

[0088] The numbers "0", "1", and "2" displayed on the input data 810 are only the channel numbers of the input data 810, and the numbers themselves do not represent the information indicated by the data. In other words, the number "0" is an element included in the first channel of the input data 810, the number "1" is an element included in the second channel of the input data 810, and the number "2" is an element included in the third channel of the input data 810. Each element can represent unique information.

[0089] The processor 420 manipulates the input data 810 to generate the first manipulated data 820. For example, the processor 420 may generate the first manipulated data 820 by adding at least one channel 821, 822, 823 to the input data 810. Here, the at least one channel 821, 822, 823 to be added may be configured by zero.

[0090] The processor 420 may manipulate the input data 810 based on the number of first channels included in the input data and the number of second channels included in the hardware. Here, the hardware refers to the number of channels of an operator included in the processor 420. For example, the processor 420 may determine the number of added channels 821, 822, and 823 according to the following equation 1.

[0091] Equation 1:

[0092]

[0093] In Equation 1, N channel refers to the maximum value of the number of tracks included in each of channels 821, 822, and 823, N lane refers to the number of channels of the operator, and N input Refers to the number of channels of input data 810.

[0094] For example, assume that the input data 810 includes three channels and the operator that processes the input data 810 includes nine channels. In this case, the processor 420 converts N channel The identification is 2. The processor 420 generates the first manipulated data 820 by adding up to two lanes between channels of the input data 810. Here, the added lanes may be configured by zero.

[0095] although Figure 8 It is shown that the channels 821, 822, and 823 are configured by two lanes, but the channels are not limited thereto. According to the above example, each of the channels 821, 822, and 823 may include one or two lanes, respectively.

[0096] The processor 420 manipulates the first manipulated data 820 to generate the second manipulated data 830. For example, the processor 420 may shift the elements of each column of the plurality of columns included in the first manipulated data 820 according to a specified rule. Here, the specified rule refers to a rule that shifts the elements of each column of the plurality of columns by a specified size in the same direction. In addition, the specified rule may be periodically applied to the plurality of columns.

[0097] refer to Figure 8, the processor 420 may shift the elements of each column of the columns col0-col 5 included in the first manipulated data 820 according to a specified rule. The specified rule may be a rule for shifting the elements of each column of the multiple columns col 0-col 5 by a specified size in the same direction. Here, the specified size may be adaptively changed by the processor 420 according to the form of the sparsity of the first manipulated data 820, and the size of the shift to be applied to each column of the multiple columns col 0-col 5 may be changed entirely. For example, the processor 420 may generate the second column col 1 of the second manipulated data 830 by shifting the activation included in the second column col 1 of the first manipulated data 820 by one row. In addition, the processor 420 may generate the fifth column col 4 of the second manipulated data 830 by shifting the activation included in the fifth column col 4 of the first manipulated data 820 by two rows. In addition, the processor 420 may shift or not shift activations of other columns col 0, 2, 3, 5 of the first manipulated data 820 according to the form of sparsity of the first manipulated data 820 .

[0098] In addition, the specified rules can be applied periodically to multiple columns col 0-col 5. Figure 8 As shown in , the processor 420 may periodically apply the shift rule of “0-1-2-1-2-0” to the next data of the first manipulated data 820. For example, the period may be the same as the size of the core data, but is not limited thereto.

[0099] Return to reference Figure 5 , in operation 520 , the processor 420 rearranges the manipulated data based on the sparsity of the manipulated data.

[0100] Sparsity refers to whether there is a data gap or the state of the data included in the gap. For example, valid information can be represented by a non-zero number. Here, valid information refers to data on which a meaningful convolution operation can be performed. Generally, information can be represented by a number, so valid information can represent data as a non-zero number. In other words, meaningless information can be represented as zero data.

[0101] Therefore, data represented as zeros means meaningless information, which can also be understood as blank data (ie, no data). Therefore, the processor 420 identifies the sparsity of the manipulated data in the same way as the processor 420 identifies the distribution of zeros in the manipulated data.

[0102] The processor 420 can rearrange the manipulated data in various ways. For example, the processor 420 can shift at least one element included in the manipulated data from the first position of the first column to the second position of the second column. Here, the first column refers to the column located before the element is shifted, and the second column refers to the column located after the element is shifted. At this time, the first position of the first column and the second position of the second column can be positions corresponding to each other, or can be different positions.

[0103] In operation 530, the processor 420 processes the rearranged data to generate output data. For example, the processor 420 may generate output data by performing a convolution operation using the rearranged data. An example of the processor 420 performing a convolution operation is described above with reference to Figure 2 described.

[0104] In the following, reference will be made to FIG. 9A to FIG. 14 An example in which the processor 420 rearranges the manipulated data is described below.

[0105] Fig.9A and Fig. 9B is a diagram showing an example of a processor rearranging manipulated data.

[0106] Fig.9A and Fig. 9B There are shown manipulated data 910 and 920 and core data 930 and 940. If the manipulated data 910 and 920 are rearranged, the core data 930 and 940 may also be rearranged to correspond to the rearranged data.

[0107] For example, the processor 420 rearranges the kernel data 930 and 940 so that weights corresponding to activations input to the operators 950 and 960 are input to the operators 950 and 960. Then, the processor 420 inputs the weights into the operators 950 and 960 according to the rearranged kernel data 930 and 940. Therefore, even for the rearranged data, accurate operation results can be output from the operators 950 and 960.

[0108] refer to Fig.9A , the processor 420 inputs some of the manipulated data 910 to the operator 950. For example, the processor 420 may input the activations in the window 970 included in the manipulated data 910 to the operator 950. At this time, the processor 420 may apply a specified rule to the activations included in the window 970 to input the maximum activation to the operator 950.

[0109] The processor 420 may shift at least one element included in the window 970 from a first position in the first column to a second position in the second column. Here, the first position and the second position may be positions corresponding to each other. For example, the processor 420 may identify blanks in columns col 0 and col 1 in the window 970, and assign activation of column col 1 to the blank of column col 0. Fig.9A , we can see that activations 0, 1, and 2 of column col 1 are shifted to the same positions in column col 0.

[0110] The processor 420 inputs the rearranged data (activation) to the input layer 951 of the operator 950 according to the above manner. When the column col 0 of the manipulated data 910 is compared with the input layer 951, the number of blanks of the input layer 951 is less than the number of blanks of the column col 0. This means that the blank includes data 0, so no matter what the value of the weight corresponding to the blank is, the output is zero. Therefore, the more the number of blanks included in the input layer 951 (that is, the more the number of zeros included in the input layer 951), the more unnecessary operations.

[0111] refer to Fig. 9B , the size of window 971 is larger than the size of window 970. In other words, the processor 420 may set various sizes of windows 970 and 971. For example, the processor 420 may set the sizes of windows 970 and 971 to correspond to the number of times the input data is manipulated, but is not limited thereto. Fig.9A and Fig. 9B When compared, the sizes of windows 970 and 971 are different, but the manner in which the manipulated data 910 and 920 are rearranged is the same.

[0112] Fig.10 is a diagram showing another example of a processor rearranging manipulated data.

[0113] Fig.10 1010 and core data 1020 are shown. If the manipulated data 1010 is rearranged, the core data 1020 may also be rearranged to correspond to the rearranged data, as described above with reference to FIG. Fig.9A and Fig. 9B described.

[0114] The processor 420 inputs some of the manipulated data 1010 to the operator 1030. For example, the processor 420 may input activations included in the window 1040 of the manipulated data 1010 to the operator 1030.

[0115] The processor 420 may shift at least one element included in the window 1040 from a first position of the first column to a second position of the second column. Here, the first position and the second position may be different from each other. For example, the processor 420 may identify the blanks of columns col 0 and col 1 in the window 1040, and may assign the activation of column col 1 to the blank of column col 0. Fig.10 , you can see that activations 0, 1, and 3 of column col 1 are shifted horizontally to column col 0.

[0116] In the above manner, the processor 420 inputs the rearranged activations to the input layer 1031 of the operator 1030. When the column col 0 of the window 1040 is compared with the input layer 1031, there is a blank in the column col 0, but there is no blank in the input layer 1031. Therefore, the processor 420 can minimize the number of unnecessary operations performed by the operator 1030.

[0117] As referenced above FIG. 9A to FIG. 10 As described, the processor 420 is shown as being configured to be operable by applying Fig.9A and Fig. 9B method or Fig.10 The processor 420 may identify the sparsity of the manipulated data 910, 920 and 1010 or the core data 930, 940 and 1020, and Fig.9A and Fig. 9B as well as Fig.10 At least one of the methods is adaptively applied to the manipulated data 910 , 920 and 1010 and / or the core data 930 , 940 and 1020 .

[0118] Fig.11 is a diagram showing another example of a processor rearranging manipulated data.

[0119] Fig.11 Manipulated data 1110 and core data 1120 are shown. Some of the manipulated data 1110 include blanks. Here, blanks can be interpreted as having no valid information, for example, the activation corresponding to a blank can be zero. In addition, Fig.11 In the embodiment, blank is included only in the manipulated data 1110, but is not limited thereto. In other words, 0 may also be included in at least one of the weights, and at least one of the weights is included in the core data 1120.

[0120] The processor 420 may rearrange the manipulated data 1110 in a manner based on the sparsity of the manipulated data 1110. For example, the processor 420 may rearrange the multiple rows row 0-row 5 based on the number of zeros (i.e., the number of blanks) included in each of the multiple rows row 0-row 5, wherein the multiple rows row 0-row 5 are included in the manipulated data.

[0121] For example, with reference to manipulated data 1110 and rearranged data 1111, processor 420 can arrange row 2 including most zeros and row 0 including least zeros as adjacent among multi-row row 0-row 5 of manipulated data 1110. In a similar manner, processor 420 can arrange row 5 including most zeros (the same quantity as row 2) and row 1 including the second least zeros as adjacent. In addition, processor 420 can arrange row 4 including the second most zeros and row 3 including the second least zeros as adjacent among multi-row row 0-row 5 of manipulated data 1110. In this way, processor 420 can generate rearranged data 1111 by rearranging multi-row row 0-row 5 of manipulated data 1110 based on the quantity including zeros (that is, the quantity of blanks).

[0122] In addition, the processor 420 may be configured according to the above reference FIG. 9A to FIG. 10 The described method reorders the activations included in the window 1140 among the reordered data 1111, and the reordered activations may be input to the operator 1130. In addition, the processor 420 may also apply a window 1150 having the same size to the kernel data 1120 to reorder the weights included in the window 1150, and input the reordered weights to the operator 1130.

[0123] Fig.12 is a diagram showing another example of a processor rearranging manipulated data.

[0124] Fig.12 Manipulated data 1210 and core data 1220 are shown. Some of the manipulated data 1210 include blanks. In addition, blanks are shown as only included in Fig.12 In the manipulated data 1210 in, but not limited to this. In other words, 0 may also be included in at least one of the weights, and at least one of the weights is included in the core data 1220.

[0125] The processor 420 may rearrange the manipulated data 1210 in a manner based on the sparsity of the manipulated data 1210. For example, the processor 420 may shift elements of each of the columns col 0 to col 5 included in the manipulated data 1210 according to the following method.

[0126] For example, the processor 420 may shift the elements of each column in the multi-column col 0 to col5 by a specified size in the same direction. Here, the specified size may be adaptively changed by the processor 420 according to the form of the sparsity of the manipulated data 1210, and the shift size to be applied to each column in the multi-column col 0 to col 5 may be different from each other. For example, with reference to the manipulated data 1210 and the rearranged data 1211, the processor 420 may generate the second column col 1 of the rearranged data 1211 by shifting the activation included in the second column col1 of the manipulated data 1210 by one space (e.g., one space downward). In addition, the processor 420 may generate the fifth column col 4 of the rearranged data 1211 by shifting the activation included in the fifth column col 4 of the manipulated data 1210 by two spaces (e.g., two spaces downward). In addition, depending on the form of the sparsity of the manipulated data 1210, the processor 420 may not shift the activations of the other columns col0, col2, col3, and col5 of the manipulated data 1210.

[0127] In addition, the above method can be periodically applied to multiple columns col 0 to col 5. Fig.12 As shown in , the processor 420 may periodically apply the shift rule of "0-1-0-0-2-0" to the data to be input after the manipulated data 1210. For example, the period may be the same as the size of the core data 1220, but is not limited thereto. Through this process, the processor 420 may prevent unnecessary convolution operations from being performed.

[0128] In addition, the processor 420 also rearranges the core data 1220 to correspond to the rearranged data 1211. For example, the processor 420 rearranges the core data 1220 so that the weight to be input to the operator for the activation operation is correctly input to the operator. In addition, the processor 420 inputs the weight to the operator according to the rearranged core data. Therefore, even for the rearranged data 1211, an accurate operation result can be output from the operator.

[0129] If the core data 1220 is rearranged, the processor 420 rearranges the manipulated data 1210 in the same manner as described above, and inputs the rearranged data to the operator.

[0130] Fig.13is a diagram showing another example of a processor rearranging manipulated data.

[0131] Fig.13 Manipulated data 1310 and core data 1320 are shown. Some of the manipulated data 1310 include blanks. In addition, blanks are shown as only included in Fig.13 In the manipulated data 1310 in, but not limited to this. In other words, 0 may also be included in at least one of the weights, and at least one of the weights is included in the core data 1320.

[0132] The processor 420 may rearrange the manipulated data 1310 in a manner based on the sparsity of the manipulated data 1310. For example, the processor 420 may shift the first element (activation) of the column col 1 included in the manipulated data 1310 to the position of the last element (activation) of the column col 0 adjacent to the column col 1.

[0133] The first positions of columns col 1 and col 0 include valid information. In addition, the last position of column col 0 does not include valid information. In this case, processor 420 can shift the element of the first position of column col 1 to the last position of column col 0. Through this process, processor 420 can prevent unnecessary convolution operations from being performed.

[0134] When the manipulated data 1310 is rearranged, the above reference FIG. 9A to FIG. 12 The kernel data 1320 is rearranged as described.

[0135] Fig.14 is a diagram showing another example of a processor rearranging manipulated data.

[0136] Fig.14 The manipulated data 1410 is shown. Some of the manipulated data 1410 include blanks. In particular, some columns col 1 to col 3 of the manipulated data 1410 are all configured only by blanks.

[0137] The processor 420 may rearrange the manipulated data 1410 in a manner based on the sparsity of the manipulated data 1410. For example, the processor 420 may rearrange the manipulated data 1410 so that processing of columns col 1 to col 3 including only zeros among the plurality of columns col 0 to col 5 included in the manipulated data 1410 may be omitted.

[0138] For example, the processor 420 may omit columns col 1 to col 3 from the manipulated data 1410 and generate the rearranged data 1420 using only the remaining columns col 0, col 4, and col 5. In addition, the processor 420 records that the columns col 1 to col 3 are omitted in the memory 410. Through this process, the processor 420 may prevent unnecessary convolution operations from being performed.

[0139] At the same time, when the manipulated data 1410 is rearranged, the above reference FIG. 9A to FIG. 12 The nuclear data were rearranged as described.

[0140] As described above, the apparatus 400 manipulates the input data based on the input data and the configuration of the hardware for processing the input data. Therefore, the apparatus 400 can process the data without an idle channel in the operator.

[0141] In addition, the apparatus 400 rearranges the manipulated data based on the sparsity of the manipulated data. Therefore, the apparatus 400 can output a valid result without performing unnecessary operations, thereby reducing the total number of operations while outputting a desired result.

[0142] The above method can be performed by a program executable in a computer, and can be implemented by a general-purpose digital computer that executes the program by using a computer-readable recording medium. In addition, the structure of the data used in the above method can be recorded on a computer-readable recording medium by various means. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) or an optical reading medium (e.g., CD-ROM, DVD, etc.).

[0143] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of the present application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered to be illustrative only and not for limiting purposes. The description of the features or aspects in each example is considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents are interpreted as being included in the present disclosure.

Claims

1. A method for processing data, the method comprising: manipulating the input data based on a comparison result of a first number of input channels of the input data and a second number of channels of the operator to generate manipulated data; Rearranging the manipulated data based on sparsity of the manipulated data to generate rearranged data; as well as processing the rearranged data to generate output data, The manipulating the input data includes adding at least one input channel to the input data based on a comparison result that a first number of input channels of the input data is less than a second number of channels of the operator, so that there are no idle operator channels.

2. The method of claim 1 , wherein manipulating the input data comprises: At least one channel configured by zero is added to the input data based on the configuration of hardware for processing the input data. 3 . The method of claim 1 , wherein the input data is manipulated based on a number of first channels included in the input data and a number of second channels included in hardware for processing the input data. 4 . The method of claim 3 , wherein the input data is manipulated based on a value obtained by dividing the number of the second channels by the number of the first channels.

5. The method of claim 4, wherein manipulating the input data comprises: adding n channels between the first channels, each of the n channels being configured by zeros, and Here, n is a natural number that is less than or equal to the value obtained by dividing the number of the second channels by the number of the first channels.

6. The method of claim 1 , wherein manipulating the input data comprises: Elements of one or more columns included in the input data are shifted according to a specified rule.

7. The method according to claim 6, wherein the specified rules include: shifts the elements of the column or columns by a specified amount in the same direction, and The specified rule is periodically applied to the one or more columns.

8. The method according to claim 1, wherein rearranging the manipulated data comprises: At least one element included in the manipulated data is shifted from a first position of a first column including the at least one element to a second position of a second column. 9 . The method of claim 8 , wherein the first position of the first column and the second position of the second column correspond to each other. 10 . The method of claim 8 , wherein the first position of the first column and the second position of the second column are different from each other. 11 . A non-transitory computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

12. A device for processing data, the device comprising: a memory storing at least one program; as well as A processor is configured to execute the at least one program to perform the following operations: manipulating the input data based on a comparison result of a first number of input channels of the input data and a second number of channels of the operator to generate manipulated data; Rearranging the manipulated data based on sparsity of the manipulated data to generate rearranged data; as well as processing the rearranged data to generate output data, The manipulating the input data includes adding at least one input channel to the input data based on a comparison result that a first number of input channels of the input data is less than a second number of channels of the operator, so that there are no idle operator channels. 13 . The apparatus according to claim 12 , wherein the processor is configured to add at least one channel configured by zero to the input data based on a configuration of hardware for processing the input data. 14 . The apparatus of claim 12 , wherein the processor is configured to manipulate the input data based on a number of first channels included in the input data and a number of second channels included in hardware for processing the input data. 15 . The apparatus of claim 14 , wherein the processor is configured to manipulate the input data based on a value obtained by dividing the number of the second channels by the number of the first channels.

16. The apparatus of claim 15, wherein the processor is configured to: add n channels between the first channels, each of the n channels being configured by zeros, and Here, n is a natural number that is less than or equal to the value obtained by dividing the number of the second channels by the number of the first channels. 17 . The apparatus according to claim 12 , wherein the processor is configured to shift elements of one or more columns included in the input data according to a specified rule.

18. The apparatus according to claim 17, wherein the specified rules include: shifts the elements of the column or columns by a specified amount in the same direction, and The specified rule is periodically applied to the one or more columns.

19. The apparatus of claim 12, wherein the processor is configured to shift at least one element included in the manipulated data from a first position of a first column including the at least one element to a second position of a second column.

20. The device of claim 19, wherein the first position of the first column and the second position of the second column correspond to each other.

21. The device of claim 19, wherein the first position of the first column and the second position of the second column are different from each other.

22. An apparatus for processing data, comprising: Memory; as well as A processor is configured to execute at least one program stored in the memory to perform the following operations: generating first data by manipulating the input data based on a comparison result of a first number of input channels of the input data and a second number of channels of the operator; generating second data by rearranging the first data based on the validity of elements included in the first data; as well as performing a convolution operation on the second data to generate output data, The manipulating the input data includes adding at least one input channel to the input data based on a comparison result that a first number of input channels of the input data is less than a second number of channels of the operator, so that there are no idle operator channels.

23. The apparatus of claim 22, wherein generating the second data comprises: At least one invalid element in the first data is replaced with a valid element.

24. The apparatus of claim 23, wherein the at least one invalid element is zero and the valid elements are non-zero numbers.

Images