Electronic device and control method thereof

Abstract

An electronic device and a method thereof are provided for performing deep learning. The electronic device includes: a memory configured to store target data and core data; and a processor including a plurality of processing elements arranged in a matrix shape. The processor is configured to: input a first non-zero element of a plurality of first elements included in the target data to each of the plurality of processing elements, and sequentially input a second non-zero element of a plurality of second elements included in the core data to each of the plurality of first processing elements included in a first row of the plurality of processing elements. Each of the plurality of first processing elements is configured to perform an operation between the input first non-zero element and the input second non-zero element based on depth information of the first non-zero element and depth information of the second non-zero element.

Description

Technical Field

[0001] The present disclosure generally relates to an electronic device and a control method thereof, and more particularly, to an electronic device for performing a convolution operation and a control method thereof. Background Art

[0002] A touch sensing device such as a touch pad can provide an input method using its own body without a separate input device such as a mouse or a keyboard. The touch sensing device is generally applied to portable electronic devices such as notebooks, where it is difficult to use a separate input device.

[0003] In recent years, artificial intelligence systems that achieve human-level intelligence have been used in various fields. Unlike existing rule-based intelligent systems, in artificial intelligence systems, machines learn, make decisions, and become smarter. Artificial intelligence systems are becoming more and more common, and existing rule-based intelligent systems are being replaced by these deep learning-based types of artificial intelligence systems.

[0004] Artificial intelligence technologies include machine learning (e.g., deep learning) and basic technologies that utilize machine learning.

[0005] Machine learning includes algorithmic technologies that can classify / learn features of input data by themselves. Basic technologies use machine learning algorithms such as deep learning to simulate functions such as recognition and judgment of the human brain. Basic technologies include technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control.

[0006] Artificial intelligence technology can be applied to language understanding, visual understanding, reasoning / prediction, knowledge representation and motion control.

[0007] Language understanding is a technology for recognizing, applying / processing human language / characters, and includes natural language processing, machine translation, dialogue systems, query response, speech recognition / synthesis, etc. Visual understanding is a technology for recognizing and processing objects according to human vision, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning / prediction is a technology for determining information, logical reasoning, and predicting information, including knowledge / probability-based reasoning, optimization prediction, preference-based planning, and recommendation.

[0008] Knowledge representation is a technology used to automatically transform human experience information into knowledge data, including knowledge construction (data generation / classification) and knowledge management (data utilization). Motion control is a technology used to control the autonomous driving of vehicles and the movement of robots, including motion control (navigation, collision, driving) and manipulation control (behavior control).

[0009] In particular, a convolutional neural network (CNN) has a structure for learning two-dimensional data or three-dimensional data and can be trained by a back-propagation algorithm. CNN is widely used in various application fields such as object classification, object detection, etc.

[0010] Most operations of CNN are convolution operations, and most convolution operations include multiplication processing between input data. However, the target data (e.g., image) and kernel data as input data may include multiple zeros, so it is not necessary to perform multiplication operations in these cases.

[0011] For example, when at least one of the input data is zero in the multiplication operation between the input data, the multiplication result is zero. That is, if at least one of the input data is zero, it can be known that the result is zero even if the multiplication operation is not performed. Therefore, the operation cycle can be shortened by omitting unnecessary multiplication operations represented as processing data sparsity.

[0012] However, in the related art, methods have been developed only for processing data sparsity when a plurality of processing elements are implemented in the form of a one-dimensional array. Therefore, a method for processing data sparsity when a plurality of processing elements are implemented in the form of a two-dimensional array is needed. Summary of the invention

[0013] Technical issues

[0014] The present disclosure has been made to address the above-mentioned problems and disadvantages, and to provide at least the advantages described below.

[0015] Solutions to the problem

[0016] Therefore, an aspect of the present disclosure is to provide an electronic device and a control method thereof that omits unnecessary operations in a convolution operation process to increase an operation speed.

[0017] Another aspect of the present disclosure is to provide an electronic device and a control method thereof that can improve the speed of a convolution operation by omitting an operation of a portion of target data and a portion of kernel data according to zeros included in the target data.

[0018] According to one aspect of the present disclosure, an electronic device is provided for performing deep learning. The electronic device includes: a memory configured to store target data and kernel data; and a processor configured to include a plurality of processing elements arranged in a matrix shape, and the processor is configured to: input a first non-zero element of a plurality of first elements included in the target data to each of the plurality of processing elements, and sequentially input a second non-zero element of a plurality of second elements included in the kernel data to each of a plurality of first processing elements included in a first row of the plurality of processing elements, wherein each of the plurality of first processing elements is configured to perform an operation between the input first non-zero element and the input second non-zero element based on depth information of the first non-zero element and depth information of the second non-zero element.

[0019] According to another aspect of the present disclosure, a method is provided for controlling an electronic device to perform deep learning. The method includes: inputting a first non-zero element of a plurality of first elements included in target data to each of a plurality of processing elements; sequentially inputting a second non-zero element of a plurality of second elements included in kernel data to each of a plurality of first processing elements included in a first row of the plurality of processing elements; and performing an operation between the input first non-zero element and the input second non-zero element based on depth information of the first non-zero element and depth information of the second non-zero element.

[0020] Advantageous Effects of the Invention

[0021] According to various embodiments of the present disclosure as described above, the electronic device may increase the speed of a convolution operation by omitting an operation of a portion of target data and a portion of kernel data according to zeros included in the target data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1a and 1b illustrates a convolution operation between three-dimensional input data according to an embodiment;

[0024] Figure 2 An electronic device according to an embodiment is shown;

[0025] Figure 3 illustrates a plurality of processing elements according to an embodiment;

[0026] Figures 4a to 4d A method for inputting non-zero elements in target data and kernel data according to an embodiment is shown;

[0027] Figure 5a to Figure 5mshows an operation cycle of a processing element according to an embodiment;

[0028] Figure 6a and Figure 6b A method for handling data sparsity of kernel data according to an embodiment is shown.

[0029] Figure 7a and Figure 7b A method for processing data sparsity of target data according to an embodiment is shown.

[0030] Figure 8 illustrates a processing element according to an embodiment; and

[0031] Fig. 9 is a flowchart illustrating a method of controlling an electronic device according to an embodiment. DETAILED DESCRIPTION

[0032] Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that there is no intention to limit the present disclosure to the specific form disclosed herein; on the contrary, the present disclosure should be interpreted as covering various modifications, equivalents and / or substitutions of the embodiments of the present disclosure.

[0033] When describing the accompanying drawings, like reference numerals may be used to designate like constituent elements. For the sake of clarity and conciseness, detailed descriptions of known functions or configurations will be omitted.

[0034] Figure 1a and 1b A convolution operation between three-dimensional input data according to an embodiment is shown. The convolution operation is an operation performed with a very high weight in deep learning, wherein the convolution operation highlights the characteristics corresponding to the core data from the target data through the operation of the target data and the core data.

[0035] Reference Figure 1a , Figure 1a The left side of shows an example of three-dimensional target data (feature map data), Figure 1a An example of three-dimensional core data is shown on the right side of For example, the target data is three-dimensional data including four rows and four columns and a depth of five, while the core data is three-dimensional data including two rows and two columns and a depth of five.

[0036] Reference Figure 1b , which shows that according to Figure 1a The output data of the convolution operation of the target data and the kernel data is two-dimensional data including three rows and three columns.

[0037] In the output data, Out11 can be calculated using equation (1).

[0038] Out11=F11,1×A,1+F11,2×A,2+F11,3×A,3+F11,4×A,4+F11,5×A,5+F12,1×B,1+F12,2×B,2+F12,3×B,3+F12,4×B,4+F12,5×B, 5+F21,1×D,1+F21,2×D,2+F21,3×D,3+F21,4×D,4+F21,5×D,5+F22,1×C,1+F22,2×C,2+F22,3×C,3+F22,4×C,4+F22.5×C,5…(1)

[0039] In equation (1), the left side of the comma of F11,1 represents the row and column of the target data, and the right side of the comma of F11,1 represents the depth of the target data. For example, F21,3 represents the second row, the first column, and the third depth of the target data, and the remaining target data are displayed in the same manner. The left side of the comma of A,1 represents the row and column of the core data, and the right side of the comma represents the depth of the core data. For example, D,4 represents the second row, the first column, and the fourth depth of the core data, and the remaining core data are displayed in the same manner. In the following, for ease of description, the above symbols are used.

[0040] The rest of the output data may be calculated by operating on the same core data and other rows and columns of the target data. For example, Out23 in the output data may be calculated by operating on data included in all depths of F23, F24, F33, and F34 from the target data and the core data.

[0041] As described above, in order to perform a convolution operation between three-dimensional input data, the depth of the three-dimensional input data must be the same. In addition, even if the input data is three-dimensional data, the output data can be changed to two-dimensional data.

[0042] in addition, Figure 1b The result of omitting the operation on the contour pixels of the target data is shown, and when the operation on the contour pixels is added, another type of output data can be generated.

[0043] In the following description, for the convenience of description, individual data constituting target data (such as F11,1, F11,2, F11,3, F11,4, F11,5, F21,1, ..., F44,4, and F44,5) are described as first elements; individual data constituting core data (such as A,1, A,2, A,3, A,4, B,1, ..., C,4, D,1, D,2, D,3,, D,4) are described as second elements. In addition, in the following drawings, Figure 1a and Figure 1b The reference direction of rows, columns, and depths shown in is the same.

[0044] Figure 2 An electronic device according to an embodiment is shown.

[0045] Reference Figure 2 , the electronic device 100 includes a memory 110 and a processor 120 .

[0046] The electronic device 100 can perform deep learning, i.e., convolution operations. For example, the electronic device 100 can be a desktop personal computer (PC), a notebook, a smart phone, a tablet PC, a server, etc. Alternatively, the electronic device 100 can be a system itself in which a cloud computing environment is built. However, the present disclosure is not limited thereto, and the electronic device 100 can be any device capable of performing convolution operations.

[0047] The memory 110 may store target data, core data, etc. The target data and core data may be stored to correspond to the type of the memory 110. For example, the memory 110 may include a plurality of two-dimensional cells, and three-dimensional target data and core data may be stored in the plurality of two-dimensional cells.

[0048] The processor 120 may identify the data stored in the plurality of two-dimensional cells as three-dimensional target data and core data. For example, the processor 120 may identify the data stored in cells 1 to 25 in the plurality of cells as data of a first depth of the target data, and identify the data stored in cells 26 to 50 in the plurality of cells as data of a second depth of the target data.

[0049] The core data may be generated by the electronic device 100, or may be generated and received by an external electronic device (ie, not the electronic device 100). The target data may be information received from the external electronic device.

[0050] The memory 110 may be implemented as a hard disk, a nonvolatile memory, a volatile memory, or the like.

[0051] The processor 120 generally controls the operation of the electronic device 100 .

[0052] The processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON), but is not limited thereto, and may include at least one of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), and an ARM processor. The processor 120 may be implemented in the format of a system on chip (SoC), a large-scale integration (LSI) in which a processing algorithm is embedded, or a field programmable gate array (FPGA).

[0053] The processor 120 may include a plurality of processing elements arranged in a matrix form, and may control operations of the plurality of processing elements.

[0054] Figure 3 A number of processing elements are shown in accordance with an embodiment.

[0055] Reference Figure 3 , multiple processing elements (PEs) are arranged in a matrix form, and data can be shared between adjacent processing elements. Figure 3 It is shown that data is transmitted from the upper side to the lower side, but the present disclosure is not limited thereto, and data may be transmitted from the lower side to the upper side.

[0056] Each of the plurality of processing elements includes a multiplier and an arithmetic logic unit (ALU). The ALU may include at least one adder. Each of the plurality of processing elements may use the multiplier and the ALU to perform arithmetic operations. In addition, each of the plurality of processing elements may include a plurality of register files.

[0057] The processor 120 may input a first non-zero element among a plurality of first elements included in the target data to each of the plurality of processing elements. For example, the processor 120 may identify a first non-zero element (i.e., an element that is not zero) from the target data stored in the memory 110, and input the identified first non-zero element to the plurality of processing elements. That is, the processor 120 may extract only the first non-zero element from the target data stored in the memory 110 in real time.

[0058] Alternatively, before inputting the first non-zero element to the plurality of processing elements, the processor 120 may extract only the first non-zero element from the target data and store the first non-zero element in the memory 110. The memory 110 may store the target data and the extracted first non-zero element. The processor 120 may directly input the extracted first non-zero element to the plurality of processing elements. The processor 120 may identify the corresponding processing element among the plurality of processing elements based on the row information and column information of the first non-zero element, and input the first non-zero element to the identified processing element.

[0059] For example, if the first non-zero element is in the first row and the first column, the processor 120 may be configured to input the first non-zero element to a first processing element among the plurality of processing elements, and if the first non-zero element is in the second row and the second column, the first non-zero element may be input to a second processing element among the plurality of processing elements. The first non-zero element belonging to the first row and the first column may include a plurality of elements having different depths, and the processor 120 may input the plurality of first non-zero elements belonging to the first row and the first column to each of the plurality of register files of the first processing element.

[0060] The processor 120 may input the first non-zero element into a corresponding register file identified based on the depth information of the first non-zero element among a plurality of register files included in the processing element. The processing element may include a plurality of register files corresponding to each of the depths of the target data.

[0061] For example, the processing element may include a first register file corresponding to the first depth of the target data, a second register file corresponding to the second depth, ..., and an nth register file corresponding to the nth depth, and the processor 120 may input the element of the first depth in the first non-zero element belonging to the first row and the first column to the first register file included in the first processing element, and input the element of the second depth to the second register file included in the first processing element. If there is no element of the second depth in the first non-zero element belonging to the first row and the first column, the second register file included in the first processing element may not store the element. However, the present disclosure is not limited to this, and the processor 120 may sequentially input the first non-zero element into a plurality of register files included in the identified processing element without considering the depth information of the first non-zero element. For example, the processor 120 may store the depth information of the first non-zero element stored in each register file together with the first non-zero element.

[0062] If the first non-zero element belonging to the first row and the first column is a first depth element, a third depth element, and a fourth depth element, the processor 120 may sequentially input the first non-zero element to the first register file, the second register file, and the third register file. The processor 120 may store the first non-zero element stored in the first register file as an element of the first depth, store the first non-zero element stored in the second register file as an element of the third depth, and store the first non-zero element stored in the third register file as an element of the fourth depth.

[0063] The processor 120 may sequentially input a second non-zero element among the plurality of second elements included in the kernel data to each of the plurality of first processing elements included in the first row among the plurality of processing elements.

[0064] The processor 120 may identify the second non-zero element from the kernel data stored in the memory 110 and sequentially input the identified second non-zero element to each of the plurality of first processing elements. That is, the processor 120 may extract only the second non-zero element from the kernel data stored in the memory 110 in real time.

[0065] Here, the operation of sequentially inputting refers to the input order of elements in the plurality of second non-zero elements. For example, if there is a second non-zero element of the first depth, a second non-zero element of the second depth, and a second non-zero element of the third depth, the processor 120 may input the second non-zero element of the first depth to each of the plurality of first processing elements in the first cycle, input the second non-zero element of the second depth to each of the plurality of first processing elements in the second cycle, and input the second non-zero element of the third depth to each of the plurality of first processing elements in the third cycle.

[0066] Alternatively, before inputting the second non-zero element to each of the plurality of first processing elements, the processor 120 may extract only the second non-zero element from the kernel data and store the extracted second non-zero element in the memory 110. In this case, the memory 110 may store the kernel data and the extracted second non-zero element. The processor 120 may sequentially input the extracted second non-zero element to each of the plurality of first processing elements.

[0067] The plurality of first processing elements included in the first row of the plurality of processing elements may be a plurality of processing elements arranged at one corner of a matrix of the plurality of processing elements. Figure 3 The top four processing elements.

[0068] The processor 120 may sequentially input the second non-zero element to each of the plurality of first processing elements based on the row information, column information, and depth information of the second non-zero element. The processor 120 may sequentially input the second non-zero element to the plurality of first processing elements along with the depth information of the second non-zero element.

[0069] The processor 120 sequentially inputs the second non-zero elements included in one row and one column among the second non-zero elements to each of the plurality of first processing elements based on the depth. When all the second non-zero elements included in one row and one column are input to each of the plurality of first processing elements, the second non-zero elements included in a row and a column different from the one row and one column are input to each of the plurality of first processing elements.

[0070] For example, the processor 120 may sequentially input the second non-zero elements included in the first row and the second column to each of the plurality of first processing elements, and when the input of the second non-zero elements included in the first row and the first column is completed, the processor may sequentially input the second non-zero elements included in the first row and the second column to each of the plurality of first processing elements in order of depth.

[0071] The processor 120 may input one second non-zero element to each of the plurality of first processing elements, and when the cycle is changed, may input the second non-zero element in the next order to each of the plurality of first processing elements.

[0072] In addition, when the second non-zero element does not exist in one row and one column, the processor 120 inputs zero to each of the plurality of first processing elements, and when zero is input to each of the plurality of first processing elements, the processor 120 may input the second non-zero elements or zero included in different rows or columns to each of the plurality of first processing elements based on the number of second non-zero elements included in different rows and columns.

[0073] When the operation between the elements corresponding to a row and a column is completed, the accumulated result is shifted, which is why zeros are input.

[0074] When a depth in which there is no first non-zero element in all rows and columns is identified from the first non-zero elements stored in each of the plurality of processing elements, the processor 120 may omit input of the second non-zero element corresponding to the depth, and sequentially input the second non-zero elements not corresponding to the depth to each of the plurality of first processing elements.

[0075] For example, if there is no first non-zero element corresponding to the third depth among the first non-zero elements stored in each of the plurality of processing elements, the processor 120 may omit the input of the second non-zero element corresponding to the third depth among the second elements. More specifically, if the second non-zero elements belonging to the first row and the first column are elements of the first depth, the third depth, and the fourth depth, the processor 120 may input the element of the first depth among the second non-zero elements belonging to the first row and the first column to each of the plurality of processing elements, and if the cycle is changed, the processor 120 may input the element of the fourth depth among the second non-zero elements belonging to the first row and the first column to each of the plurality of first processing elements. That is, even if the element of the third depth among the second non-zero elements belonging to the first row and the first column is input to each of the plurality of first processing elements, unless there is no first non-zero element corresponding to the third depth, the operation result is zero, and the processor 120 may shorten the cycle by not inputting the element of the third depth among the second non-zero elements belonging to the first row and the first column.

[0076] Optionally, the processor 120 may further include a plurality of preliminary processing elements. When a depth at which the first non-zero element is within a predetermined number in all rows and columns is identified from the first non-zero elements stored in each of the plurality of processing elements, the processor 120 may omit input of the second non-zero element corresponding to the depth, and sequentially input the second non-zero element not corresponding to the depth to each of the plurality of first processing elements, and input the first non-zero element corresponding to the depth and the second non-zero element corresponding to the depth to the plurality of preliminary processing elements to perform an operation.

[0077] For example, among the first non-zero elements stored in each of the plurality of processing elements, if the number of first non-zero elements corresponding to the third depth is less than five, the processor 120 may omit the input of the second non-zero element corresponding to the third depth, and sequentially input the second non-zero elements not corresponding to the third depth to each of the plurality of first processing elements, and input the first non-zero element corresponding to the third depth and the second non-zero element corresponding to the third depth to the plurality of preliminary processing elements to perform the operation.

[0078] Each of the plurality of first processing elements may perform an operation on the input first non-zero element and the input second non-zero element based on the depth information of the first non-zero element and the depth information of the second non-zero element.

[0079] The remaining processing elements of the plurality of processing elements may receive the second non-zero element from the adjacent processing element. Each of the remaining processing elements may perform an operation between the input first non-zero element and the input second non-zero element based on the depth information of the first non-zero element and the depth information of the second non-zero element.

[0080] The first non-zero element and the second non-zero element may be input to each of the plurality of processing elements on a cycle-by-cycle basis. In this case, each of the plurality of processing elements may perform an operation between the first non-zero element and the second non-zero element input on a cycle-by-cycle basis based on respective depth information.

[0081] Alternatively, the first non-zero element may be preliminarily input to a plurality of processing elements at a time, and the second non-zero element may be input to each of the plurality of processing elements in each cycle. In this case, each of the plurality of processing elements may perform an operation between the pre-stored first non-zero element and the second non-zero element input in accordance with the cycle based on the respective depth information.

[0082] When the operation between the non-zero elements in the plurality of first processing elements is completed, the processor 120 may control the plurality of processing elements to shift the second non-zero elements input to the plurality of first processing elements to each of the plurality of second processing elements included in the second row. When the operation between the non-zero elements in the plurality of second processing elements is completed, the processor 120 may control the plurality of processing elements to shift the second non-zero elements shifted to the plurality of second processing elements to each of the plurality of third processing elements included in the third row among the plurality of processing elements.

[0083] When the second non-zero element input to each of the plurality of processing elements is included in the same row and the same column as the second non-zero element used in the operation just performed before, the processor 120 may add the operation result of the second non-zero element input to the previous operation result, and store the accumulated operation result in one of the plurality of register files. Here, the plurality of register files may include a register file for adding and storing the plurality of register files storing the first non-zero element and the operation result.

[0084] When the second non-zero element input to each of the plurality of processing elements and the second non-zero element used in the operation just performed before are not included in the same row and the same column, the processor 120 may shift the operation result stored in one of the plurality of register files in each of the plurality of processing elements to an adjacent processing element, accumulate the operation result of the input second non-zero element and the shifted operation result, and store the accumulated operation result in one of the plurality of register files.

[0085] Through the above method, the processor 120 can reduce unnecessary operations between the target data and the core data.

[0086] Figures 4a to 4d A method for inputting non-zero elements from target data and kernel data according to an embodiment is shown.

[0087] Reference Figure 4a , Figure 4a The left side shows the three-dimensional target data, while Figure 4a The right side of shows the three-dimensional first core data and the three-dimensional second core data.

[0088] Since the kernel data are sequentially input to the plurality of first processing elements, operations can be easily performed on the plurality of kernel data.

[0089] exist Figure 4a , the first arrow direction toward the upper right end indicates the depth direction, and the second arrow direction rotating in the clockwise direction indicates the operation order of the nuclear data. When the operation of the nuclear data corresponding to the depth of A is completed, the operation of the nuclear data corresponding to the depth of B can be performed. That is, for the entire depth thereof, the operation order may be A->B->C->D.

[0090] Reference Figure 4b , Figure 4b The upper left corner shows the first row of the target data. Figure 4b The lower left end of shows the second row of the target data. The direction of the arrow indicates Figure 4a The depth direction is shown in the direction of the first arrow.

[0091] Figure 4bThe numbers shown on the left side of represent the index of the depth and the elements are not zero, while the depth without a number represents the element is 0. For example, in the first row and the first column of the target data, the elements of the first depth, the fourth depth, and the fifth depth are not zero, and the elements of the second depth and the third depth are zero.

[0092] Figure 4b The right side shows only the Figure 4b The first non-zero element on the left side of . Figure 4b As shown on the left side of , the processor 120 may identify the first non-zero element from the target data and input the identified first non-zero element into a plurality of processing elements. Optionally, the processor 120 may only extract Figure 4b The first non-zero element shown on the right side of , the extracted first non-zero element is stored separately in the memory 110, and the stored first non-zero element is extracted to input the element to a plurality of processing elements. In this case, Figure 4b As shown, the processor 120 may first extract the first non-zero element in the depth direction of F11 of the first row, and then move to the side to extract the first non-zero element in the depth direction of F12. The processor 120 may extract the first non-zero element in the depth direction of each of F13 and F14 in the same manner. The processor 120 may extract the first non-zero element of the second row in the same manner.

[0093] exist Figure 4b In the example, for the convenience of description, only the first and second rows are shown in the target data, and for the convenience of description, only the first and second rows of the target data will be described below. However, the operation for the remaining rows is the same as that for the first and second rows.

[0094] Reference Figure 4c , Figure 4c The left side of shows the first core data and the second core data in rows and columns. Figure 4c The arrow direction in the Figure 4a The depth direction is shown in the direction of the first arrow. Figure 4c The numbers shown on the left side of represent the index of the depth and the element is not zero, while the depth without a number represents the element is 0. For example, in the first row and first column of the kernel data, the elements of the first depth and the third depth are not zero, while the elements of the second depth, the fourth depth, and the fifth depth are zero.

[0095] Figure 4c The right side shows only the Figure 4c The processor 120 may identify from the kernel data such as Figure 4c The second non-zero element shown on the left side of , and the identified second non-zero element is sequentially input into the plurality of first processing elements. Optionally, the processor 120 may only extract the second non-zero element as shown in FIG. Figure 4cThe second non-zero element shown on the right side of the first non-zero element is extracted and stored separately in the memory 110, and the stored second non-zero element is extracted to be sequentially input to the plurality of first processing elements. In this case, the processor 120 may first perform the following steps: Figure 4c Extract the first non-zero element in the depth direction of A shown in , and then move to the side to Figure 4c The processor 120 extracts the second non-zero element in the depth direction of B shown in FIG. The processor 120 may extract the second non-zero element in the depth direction of each of C and D in the same manner.

[0096] For example, Figure 4d As shown in , the processor 120 may include a plurality of processing elements in a 4×4 matrix form. The four processing elements included in the first row 410 at the upper end of the plurality of processing elements are referred to as a plurality of first processing elements.

[0097] The processor 120 may input the first non-zero element included in the first row of the target data to the plurality of first processing elements. For example, the processor 120 may input the elements of the first depth, the fourth depth, and the fifth depth included in the first row and the first column of the target data to the first processing element arranged from the left among the plurality of first processing elements, input the elements of the second depth, the third depth, and the fourth depth included in the first row and the second column of the target data to the second processing element arranged from the left among the plurality of first processing elements, input the elements of the first depth, the third depth, and the fifth depth included in the first row and the third column of the target data to the third processing element arranged from the left among the plurality of first processing elements, and input the elements of the first depth, the second depth, and the fifth depth included in the first row and the fourth column of the target data to the fourth processing element arranged from the left among the plurality of first processing elements.

[0098] The processor 120 may input the first non-zero element included in the second row of the target data to four processing elements (hereinafter, referred to as a plurality of second processing elements) included in a row located below the first row 410. For example, the processor 120 may input the elements of the first depth, the second depth, the third depth, and the fourth depth included in the second row and the first column of the target data to the first processing element from the left among the plurality of second processing elements, input the elements of the fourth depth and the fifth depth included in the second row and the second column of the target data to the second processing element from the left among the plurality of second processing elements, input the elements of the third depth included in the second row and the third column of the target data to the third processing element from the left among the plurality of second processing elements, and input the elements of the second depth, the third depth, the fourth depth, and the fifth depth included in the second row and the fourth column of the target data to the fourth processing element from the left among the plurality of second processing elements.

[0099] The processor 120 may sequentially input the second non-zero elements included in the first row and the first column of the first kernel data to the plurality of first processing elements in order of depth.

[0100] The processor 120 may sequentially input the second non-zero elements included in the first row and the first column of the first core data to multiple first processing elements, sequentially input the second non-zero elements included in the first row and the second column of the first core data to multiple first processing elements, sequentially input the second non-zero elements included in the second row and the second column of the first core data to multiple first processing elements, and sequentially input the second non-zero elements included in the second row and the first column of the first core data to multiple first processing elements.

[0101] The processor 120 may sequentially input the second non-zero elements included in the first kernel data to the plurality of first processing elements, and sequentially input the second non-zero elements included in the second kernel data to the plurality of first processing elements.

[0102] For example, the processor 120 may sequentially input the elements of the first depth and the third depth included in the first row and the first column of the first core data to the plurality of first processing elements, sequentially input the elements of the first depth, the second depth, the third depth, the fourth depth, and the fifth depth included in the first row and the second column of the first core data to the plurality of first processing elements, and sequentially input the elements of the first depth, the second depth, the third depth, and the fifth depth included in the second row and the second column to the plurality of first processing elements. If the second non-zero element is not included in the second row and the first column of the first core data, the processor 120 may input zero to the plurality of first processing elements. In addition, the processor 120 may sequentially input the second non-zero element of the second core data to the plurality of first processing elements, and the input order may be the same as that of the first core data.

[0103] The processor 120 may input one second non-zero element into the plurality of first processing elements, and sequentially input another second non-zero element into the plurality of first processing elements when the cycle changes.

[0104] When the cycle changes, each of the plurality of first processing elements may shift the second non-zero element of the input to an adjacent second processing element among the plurality of second processing elements. Each of the plurality of second processing elements may shift the second non-zero element of the input to an adjacent processing element in a lower direction.

[0105] The processor 120 may input all the first non-zero elements to the plurality of processing elements in a first cycle and input the first second non-zero element to the plurality of first processing elements. Thereafter, the processor 120 may input the second second non-zero element to the plurality of first processing elements in a second cycle after the first cycle. That is, the processor 120 may input only the second non-zero element to the plurality of first processing elements in the following cycle.

[0106] Alternatively, the processor 120 may input all first non-zero elements and first non-zero elements corresponding to the plurality of first processing elements to the plurality of first processing elements in a first cycle, and input the first second non-zero element to the plurality of first processing elements. Thereafter, the processor 120 may input the first non-zero elements corresponding to the plurality of second processing elements to the plurality of second processing elements in a second cycle, and input the second second non-zero element to the plurality of second processing elements. That is, the processor 120 may input a portion of the first non-zero elements to the plurality of first processing elements in a cycle.

[0107] Figure 5a to Figure 5m The operation of the processing element according to the cycle is shown. For ease of description, reference will be made to Figures 4a to 4d The first processing elements and the second processing elements are described as follows: Figure 5a to Figure 5m . Specifically, Figure 5a to Figure 5m A plurality of first processing elements are shown on the upper side, and a plurality of second processing elements are shown on the lower side. In addition, in each processing element, the left side represents the first non-zero element, the middle represents the second non-zero element, and the right side represents the processing result.

[0108] Reference Figure 5a , Figure 5a The upper left side of shows one of the plurality of first processing elements, and the left side 510 represents the first non-zero elements of the first depth, the fourth depth, and the fifth depth included in the first row and the first column in the target data, the middle element 520 represents the second non-zero element of the first depth included in the first row and the first column in the first core data, and the right side 530 represents the operation result. However, the description of the specific operation result value is omitted in the right side 530.

[0109] like Figure 5a As shown in , the processor 120 may input the first non-zero element to the plurality of first processing elements and the plurality of second processing elements in the first cycle. However, the present disclosure is not limited thereto, and the processor 120 may input the first non-zero element to the plurality of first processing elements in the first cycle, and input the first non-zero element to the plurality of second processing elements in the second cycle. The processor 120 may input the first non-zero element corresponding to each processing element, and further description will be omitted.

[0110] The processor 120 may input the second non-zero element to the first processing element in the first cycle. Here, the input second non-zero element is the second non-zero element of the first depth included in the first row and the first column of the first kernel data.

[0111] Based on the input first non-zero element depth information and the input second non-zero element depth information, each of the plurality of first processing elements may perform an operation between the input first non-zero element and the input second non-zero element and store the operation result. For example, the input second non-zero element is an element of the first depth, and therefore, the first processing element, the third processing element, and the fourth processing element storing the first non-zero element of the first depth from the left may perform an operation between the first non-zero element and the second non-zero element. Among the plurality of first processing elements, the second processing element from the left that does not store the first non-zero element of the first depth does not perform an operation between the first non-zero element and the second non-zero element. The operation result is stored in each processing element and is not shifted to an adjacent processing element.

[0112] Since the second non-zero element is not input, the plurality of second processing elements do not perform an operation.

[0113] Reference Figure 5b , the processor 120 may input the second non-zero element to the plurality of first processing elements. Here, the input second non-zero element is the second non-zero element of the third depth included in the first row and the first column of the first kernel data.

[0114] Each of the plurality of first processing elements may shift the second non-zero element to an adjacent second processing element in the first cycle.

[0115] Each of the plurality of first processing elements may perform an inter-element operation between a first non-zero element input and a second non-zero element input. Each of the plurality of first processing elements may shift the operation result to an adjacent processing element by adding the operation result of the second cycle to the operation result of the first cycle. The reason for the shift is that all second non-zero elements included in the first row and the first column are input into the first core data. That is, the second non-zero element input in the second cycle is the last second non-zero element included in the first row and the first column of the first core data.

[0116] The shift direction is determined according to the row and column where the element is located in the first core data in the next cycle. In the third cycle, the second non-zero element of the first depth included in the first row and the second column of the first core data will be input, and the second non-zero element is on the right side of the first row and the first column of the first core data. That is, the shift direction may be to the right. If the second non-zero element of the first depth included in the second row and the first column will be input in the third cycle, this is on the lower side of the first row and the first column of the first core data, and the shift direction may be to the downward side.

[0117] Each of the plurality of second processing elements may perform an inter-element operation between a first non-zero element and a second non-zero element input by the same operation method as an operation of the plurality of first processing elements in a previous cycle.

[0118] like Figure 5c As shown, the processor 120 may input the second non-zero element to the plurality of first processing elements in the third cycle. Here, the input second non-zero element is the second non-zero element of the first depth included in the first row and the second column of the first kernel data.

[0119] Each of the plurality of first processing elements may shift the second non-zero element input in the second cycle to an adjacent second processing element. In addition, each of the plurality of second processing elements may shift the second non-zero element input in the second cycle to a processing element (not shown) adjacent to the lower side of the processing element to which the second non-zero element is input.

[0120] In other words, the plurality of first processing elements and the plurality of second processing elements may shift the second non-zero element in the previous cycle, and when the cycle changes, the element may be shifted to the lower processing element together with the input second non-zero element. Because the same operation is repeated, the description of the shift of the second non-zero element will be omitted.

[0121] Each of the plurality of first processing elements may perform an inter-element operation on the input first non-zero element and the input second non-zero element. Each of the plurality of first processing elements may add an operation result of the second cycle to an operation result of the third cycle and store the added operation result.

[0122] Each of the plurality of second processing elements may perform an inter-element operation between a first non-zero element of an input and a second non-zero element of an input input in the same operation method as an operation of the plurality of first processing elements in a previous cycle, and shift the operation result to the right.

[0123] That is, each of the plurality of second processing elements may operate in the same manner as the operation of the plurality of first processing elements in the previous cycle. Hereinafter, unless otherwise specified, the operation of the plurality of second processing elements is the same as the operation of the plurality of first processing elements in the previous cycle.

[0124] Figure 5d , Figure 5e and Figure 5f Each of shows an operation according to the input of the second non-zero elements of the second depth, the third depth, and the fourth depth included in the first row and the second column of the first kernel data. The operation is the same as the above operation, so the detailed description is omitted.

[0125] Reference Figure 5g, the processor 120 may input the second non-zero element to the plurality of first processing elements in the seventh cycle. Here, the input second non-zero element is the second non-zero element of the fifth depth included in the first row and the second column of the first kernel data.

[0126] Each of the plurality of first processing elements may perform an inter-element operation on the first non-zero element of the input and the second non-zero element of the input. Each of the plurality of first processing elements may add the operation result of the seventh cycle to the operation result of the sixth cycle and shift it to an adjacent second processing element.

[0127] As described above, in the next cycle, the second non-zero element of the first depth included in the second row and the second column of the first core data (wherein the second non-zero element corresponds to the lower side of the first row and the second column of the first core data) will be input, and the shift direction may be downward. Each of the plurality of second processing elements may perform an inter-element operation between the input first non-zero element and the input second non-zero element.

[0128] Each of the plurality of second processing elements may store the operation result shifted from the adjacent first processing element separately from the operation result in the seventh cycle. That is, the operation result shifted in the downward direction from the processing element adjacent to the upper side is not added to the operation result of the current cycle.

[0129] Reference Figure 5h , the processor 120 may input the second non-zero element to the plurality of first processing elements in the eighth cycle. The input second non-zero element is the second non-zero element of the first depth included in the second row and the second column of the first kernel data.

[0130] Each of the plurality of first processing elements may perform an inter-element operation between a first non-zero element of the input and a second non-zero element of the input.

[0131] Each of the plurality of second processing elements may perform an inter-element operation on the first non-zero element of the input and the second non-zero element of the input. Each of the plurality of second processing elements may add the operation result in the seventh cycle to the operation result in the eighth cycle, and shift the added operation result to the processing element adjacent to the lower side. However, the operation result shifted from the processing element adjacent to the upper side in the seventh cycle may be stored in each of the plurality of second processing elements as is.

[0132] Reference Figure 5i , the processor 120 may input the second non-zero element to the plurality of first processing elements in the ninth cycle. The input second non-zero element is the second non-zero element of the second depth included in the second row and the second column of the first kernel data.

[0133] Each of the plurality of first processing elements performs an inter-element operation between an input first non-zero element and an input second non-zero element, and stores an added operation result by adding an operation result of a previous cycle to an operation result of a current cycle.

[0134] Each of the plurality of second processing elements performs an inter-element operation between the first non-zero element of the input and the second non-zero element of the input, adds the operation result shifted from the processing element adjacent to the upper side in the seventh cycle to the operation result of the current cycle, and stores the added operation result.

[0135] Figure 5j and Figure 5k An operation according to input of the second non-zero elements of the third depth and the fifth depth included in the second row and the second column of the first kernel data is shown. As described above, the operation method, the addition method and the shift method are the same, and therefore, a detailed description is omitted.

[0136] However, if Figure 5k As shown in , the result of the addition operation can be shifted to the left. That is, Figure 5k The shift direction of the addition result can be Figure 5b The shift direction of the addition result is opposite.

[0137] Reference Figure 5l , the processor 120 may input zeros to the plurality of first processing elements in the twelfth cycle. Since there is no second non-zero element in the second row and the first column of the first core data, the processor 120 may input zeros to the plurality of first processing elements.

[0138] exist Figure 5l , since the second non-zero element input in the next cycle is the second non-zero element of the second core data, no shift is required. However, if the second non-zero element to be input in the next cycle is the second non-zero element of the same first core data, a shift is performed. In this case, the processor 120 inputs zero to the plurality of first processing elements, and the operation result stored in each of the plurality of first processing elements may be shifted to an adjacent processing element.

[0139] Reference Figure 5m , the processor 120 may input the second non-zero element to the plurality of first processing elements in the thirteenth cycle. The input second non-zero element is the second non-zero element of the second depth included in the first row and the first column of the second core data. The operations of the plurality of first processing elements and the plurality of second processing elements are the same as those described above.

[0140] By using Figure 5a to Figure 5m The above method shown in can perform continuous convolution operations on multiple kernel data. Here, the processor 120 can output the operation result of the first kernel data.

[0141] although Figure 5a to Figure 5m The plurality of processing elements are shown in the form of a 4×4 matrix, but the present disclosure is not limited thereto and the number of processing elements may vary.

[0142] In addition, although the target data has been described in the form of 4×4×5, the target data is not limited thereto, and the target data may be in any other form. For example, when the target data is in the form of 16×16×5 and a plurality of processing elements in the form of a 4×4 matrix are used, the processor 120 may divide the target data into four parts based on the rows and columns of the target data, and may perform a convolution operation.

[0143] Figure 6a and Figure 6b A method of processing data sparsity of kernel data according to an embodiment is shown.

[0144] If the processor 120 identifies a depth where there is no first non-zero element in all rows and columns of the first non-zero elements stored in each of the plurality of processing elements, the processor may omit input of the second non-zero element corresponding to the depth among the second elements and sequentially input the second non-zero element not corresponding to the depth to each of the plurality of first processing elements.

[0145] For example, Figure 6a As shown, the processor 120 may recognize that there is no first non-zero element corresponding to the second depth in the first non-zero elements stored in each of the plurality of processing elements. In this case, the processor 120 may remove the second non-zero element of the second depth included in the first core data and the second core data, and sequentially input the remaining second non-zero elements to the plurality of first processing elements.

[0146] The processor 120 may remove the second non-zero elements of the second depth included in the first core data and the second core data, store the remaining second non-zero elements separately in the memory 110, and sequentially extract the remaining second non-zero elements to input to the plurality of first processing elements. Alternatively, the processor 120 may sequentially extract the second non-zero elements from the first core data and the second core data, and when the second non-zero element of the second depth is identified, the second non-zero element will be skipped, and the second non-zero element that is not the second depth may be extracted, and the second non-zero element that is not the second depth may be input to the plurality of first processing elements.

[0147] Alternatively, if Figure 6b As shown in , the processor 120 may identify a depth at which there are no first non-zero elements in all rows and all columns before inputting the second non-zero element to each of the plurality of processing elements.

[0148] Figure 7a and Figure 7bA method for processing data sparsity of target data according to an embodiment is shown.

[0149] If a depth at which the first non-zero element is within a predetermined number in all rows and columns is identified from the first non-zero elements stored in each of the plurality of processing elements, the processor 120 may omit input of the second non-zero element corresponding to the identified depth among the second elements and sequentially input the second non-zero elements that do not correspond to the depth to each of the plurality of first processing elements.

[0150] For example, Figure 7a As shown in , when a second depth having less than three first non-zero elements in all rows and columns of the first non-zero elements stored in each of the plurality of processing elements is identified, the processor 120 may omit input of the second non-zero element 720 corresponding to the second depth among the second elements, and may sequentially input the second non-zero elements not corresponding to the second depth to each of the plurality of first processing elements.

[0151] In this case, the first non-zero element of the identified depth may be stored in a portion of the plurality of processing elements, but unless the second non-zero element 720 of the identified depth is input, the operation is not performed, and thus the cycle may be shortened. Figure 6a and Figure 6b The cycle is the same as shown in .

[0152] The processor 120 may also include a plurality of preliminary processing elements, and may input the first non-zero element corresponding to the identified depth and the second non-zero element corresponding to the identified depth to the plurality of preliminary processing elements to perform separate operations.

[0153] For example, Figure 7b As shown in , the processor 120 may further include a plurality of preliminary processing elements 730, and may input a first non-zero element 710 corresponding to the identified depth and a second non-zero element 720 corresponding to the identified depth to the plurality of preliminary processing elements 730 to perform separate operations.

[0154] In other words, processor 120 may use multiple processing elements to execute Figure 5a to Figure 5m , and uses a plurality of preliminary processing elements 730 to perform operations on the first non-zero element 710 corresponding to the identified depth and the second non-zero element 720 corresponding to the identified depth in parallel.

[0155] Thereafter, the processor 120 may add the operation results output from the plurality of preliminary processing elements 730 to corresponding operation results among the operation results output from the plurality of processing elements.

[0156] Figure 8 A processing element according to an embodiment is shown.

[0157] Reference Figure 8 The processing elements include a core terminal 811, a feature map terminal 812, a PSum terminal 813, a bottom Acc terminal 814, a left Acc terminal 821, a right Acc terminal 822, a Ctrl_Inst terminal 823, a left Acc terminal 831, a right Acc terminal 832, a core terminal 841, a PSum terminal 842, a bottom Acc terminal 843, a register file 850, a multiplier 860, a multiplexer 870 and an adder 880.

[0158] The processing element may receive the second non-zero element, the first non-zero element, and the data and instructions stored in the memory 110 through each of the core terminal 811, the feature map terminal 812, the Psum terminal 813, and the Ctrl_Inst terminal 823. In addition, the processing element may shift the second non-zero element to the processing element adjacent to the lower portion via the core terminal 841. In particular, the processing element may directly receive or output data to the memory 110 using the PSum terminal 813 and the PSum terminal 842.

[0159] The processing element can receive the operation result from the adjacent processing element through the bottom Acc terminal 814, the right Acc terminal 822 and the left Acc terminal 831. In addition, the processing element can shift the directly processed operation result to the adjacent processing element through the left Acc terminal 821, the right Acc terminal 832 and the bottom Acc terminal 843.

[0160] The register file 850 may store the first non-zero element input through the feature map terminal 812 and the operation result.

[0161] The multiplier 860 may perform a multiplication operation of the second non-zero element input through the core terminal 811 and the first non-zero element input from the register file 850 .

[0162] The multiplexer 870 may provide one of the following: an operation result input from an adjacent processing element, an operation result processed in the processing element, data input from the PSum terminal 813 , and data input from the register file 850 to the adder 880 .

[0163] The adder 880 may perform an addition operation of the multiplication result input from the multiplier 860 and the data input from the multiplexer 870 .

[0164] The processing element may also include a multiplexer.

[0165] Fig. 9is a flowchart illustrating a method of controlling an electronic device according to an embodiment. For example, the electronic device may include a processor that performs deep learning, a memory that stores target data and kernel data, and a plurality of processing elements arranged in a matrix form.

[0166] Reference Fig. 9 , in step S910, a first non-zero element among a plurality of first elements included in the target data is input to each of a plurality of processing elements.

[0167] In step S920 , a second non-zero element among the plurality of second elements included in the kernel data is sequentially input to each of the plurality of first processing elements included in the first row of the plurality of processing elements.

[0168] In step S930 , an operation between the input first non-zero element and the input second non-zero element is performed based on the depth information of the input first non-zero element and the depth information of the input second non-zero element.

[0169] Each of the plurality of processing elements includes a plurality of register files, and inputting the first non-zero element at step S910 may include identifying a corresponding processing element from the plurality of processing elements based on row information and column information of the first non-zero element, and inputting the first non-zero element to a corresponding register file among the plurality of register files included in the identified processing element.

[0170] The step S920 of sequentially inputting the second non-zero element may include sequentially inputting the second non-zero element to each of the plurality of first processing elements based on row information, column information, and depth information of the second non-zero element.

[0171] The step S920 of sequentially inputting the second non-zero elements may include sequentially inputting the second non-zero elements included in one row and one column of the second non-zero elements to each of the plurality of first processing elements based on the depth, and if all the second non-zero elements included in one row and one column are input to each of the plurality of first processing elements, inputting the second non-zero elements included in a row and a column different from the one row and the one column to each of the plurality of first processing elements.

[0172] In addition, step S920 of sequentially inputting the second non-zero elements includes: when there is no second non-zero element in one row and one column, inputting zero to each of the plurality of first processing elements, and if zero is input to each of the plurality of first processing elements, based on the number of second non-zero elements included in another row and another column, inputting the second non-zero element or zero included in the another row and the another column to each of the plurality of first processing elements.

[0173] The step S920 of sequentially inputting the second non-zero elements may include: when a depth at which there is no first non-zero element in all rows and columns is identified from the first non-zero elements stored in each of the plurality of processing elements, omitting input of the second non-zero elements corresponding to the depth from among the plurality of second elements, and sequentially inputting the second non-zero elements not corresponding to the depth to each of the plurality of first processing elements.

[0174] In addition, step S920 of sequentially inputting the second non-zero elements includes: when a depth at which the first non-zero element is within a predetermined number in all rows and columns is identified from the first non-zero elements stored in each of the plurality of processing elements, omitting input of the second non-zero elements corresponding to the depth among the plurality of second elements, sequentially inputting the second non-zero elements not corresponding to the depth to each of the plurality of first processing elements, and inputting the first non-zero element corresponding to the depth and the second non-zero element corresponding to the depth to the plurality of preliminary processing elements included in the processor.

[0175] When the operation between the elements is completed in the plurality of first processing elements, the second non-zero element of the input may be shifted to each of the plurality of second processing elements included in the second row. If the operation between the non-zero elements is completed in the plurality of second processing elements, the shifted second non-zero element may be shifted from the plurality of second processing elements to each of the plurality of third processing elements included in the third row.

[0176] When a second non-zero element input to each of the plurality of processing elements belongs to the same row and the same column as a second non-zero element used in a previously executed operation, the input second non-zero element and the previous operation result may be accumulated, and the accumulated result may be stored in one of the plurality of register files.

[0177] If a second non-zero element input to each of the plurality of processing elements does not belong to the same row and the same column as a second non-zero element used in an operation just performed before, an operation result stored in one of the plurality of register files of each of the plurality of processing elements may be shifted to an adjacent processing element, and the input second non-zero element may be accumulated with the shifted operation result and then stored in one of the plurality of register files.

[0178] According to various embodiments of the present disclosure as described above, the electronic device may increase the speed of a convolution operation by omitting calculation of a portion of target data and a portion of kernel data according to zeros included in the target data.

[0179] The target data and core data described above may be three-dimensional data in any form. In addition, the number of the plurality of processing elements included in the processor may also be different.

[0180] According to an embodiment of the present disclosure, the various embodiments described above may be implemented using software including instructions stored on a machine-readable storage medium that can be read by a machine (e.g., a computer). The device calls instructions stored in a storage medium, and may perform operations according to the called instructions, and may include an electronic device (e.g., an electronic device). When the instruction is executed by a processor, the processor may directly execute the function corresponding to the instruction or use other components to execute the function corresponding to the instruction under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

[0181] The machine-readable storage medium may be provided in the form of a non-transitory storage medium.

[0182] According to an embodiment of the present disclosure, the above-mentioned methods according to various embodiments may be provided in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded through an application store (e.g., Play Store TM ) Online publishing of the computer program product. For online publishing, at least a part of the computer program product may be temporarily or at least temporarily stored in a storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0183] In addition, the various embodiments described above may be implemented in a computer-readable medium such as a computer or similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. According to software implementation, embodiments such as the processes and functions described herein may be implemented in separate software modules. Each software module may perform one or more of the functions and operations described herein.

[0184] Computer instructions for performing processing operations of the device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in the non-volatile computer-readable medium, when executed by the processor of a specific device, cause the specific device to perform processing operations on the device according to the various embodiments described above. A non-transitory computer-readable medium is a medium (such as a register, cache, memory, etc.) that stores data for a short time but stores data semi-permanently and can be read by a device. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0185] In addition, each component (e.g., module or program) according to the various embodiments described above may include one or more entities, and some of the subcomponents in the above subcomponents may be omitted. These components may also be included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by each corresponding component before integration. According to various embodiments, the operations performed by modules, programs or other components may be performed in a sequential, parallel, repeated or heuristic manner, or at least some operations may be performed in a different order.

[0186] While the present disclosure has been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. An electronic device for performing deep learning, the electronic device comprising: a memory configured to store target data and core data, wherein the target data and the core data are three-dimensional data; and A processor configured to include a plurality of processing elements arranged in a matrix shape, wherein the processor is configured to: extracting only a plurality of first non-zero elements from a plurality of first elements included in the target data stored in the memory, and inputting the extracted plurality of first non-zero elements to each of the plurality of processing elements, and extracting only a second non-zero element from a plurality of second elements included in the kernel data stored in the memory, and sequentially inputting the extracted second non-zero element to each of a plurality of first processing elements included in a first row among the plurality of processing elements, Each of the plurality of first processing elements is configured to: perform only an operation between a second non-zero element input in each cycle and a first non-zero element of the plurality of first non-zero elements having the same depth information as the input second non-zero element, The processor is further configured to: when it is identified that there is no first non-zero element corresponding to a specific depth in all rows and columns of the first non-zero elements stored in each of the plurality of processing elements, omit input of the second non-zero element corresponding to the specific depth among the plurality of second elements, and sequentially input the second non-zero elements not corresponding to the specific depth to each of the plurality of first processing elements, wherein the first element corresponding to the specific depth is a zero element, wherein each of the plurality of processing elements comprises a plurality of register files, and Wherein, the processor is further configured to: Based on the row information and the column information of the first non-zero element, identifying a corresponding processing element from the plurality of processing elements, and The first non-zero element is sequentially input to the plurality of register files included in the identified processing element, and depth information of the first non-zero element is stored together with the first non-zero element.

2. The electronic device according to claim 1, wherein: The processor is further configured to sequentially input the second non-zero element to each of the plurality of first processing elements based on row information, column information, and depth information of the second non-zero element.

3. The electronic device according to claim 2, wherein: The processor is further configured to: sequentially inputting second non-zero elements included in one row and one column among the second non-zero elements to each of the plurality of first processing elements based on the depth, and When all second non-zero elements included in the one row and the one column are input to each of the plurality of first processing elements, second non-zero elements included in a row and a column different from the one row and the one column are input to each of the plurality of first processing elements.

4. The electronic device according to claim 3, wherein: The processor is further configured to: When there is no second non-zero element in the one row and the one column, inputting zero to each of the plurality of first processing elements, and When zero is input to each of the plurality of first processing elements, second non-zero elements or zeros included in a row and a column different from the one row and the one column are input to each of the plurality of first processing elements based on the number of second non-zero elements included in the row and the column different from the one row and the one column.

5. The electronic device according to claim 2, wherein: The processor also includes a plurality of preliminary processing elements, and The processor is further configured to: When a depth at which the first non-zero element is within a predetermined number in all rows and columns is identified from the first non-zero elements stored in each of the plurality of processing elements, input of a second non-zero element corresponding to the depth is omitted, and second non-zero elements not corresponding to the depth are sequentially input to each of the plurality of first processing elements, and A first non-zero element corresponding to the depth and a second non-zero element corresponding to the depth are input to a plurality of preliminary processing elements to perform an operation.

6. The electronic device according to claim 2, wherein: The processor is further configured to: When the operation between the non-zero elements in the plurality of first processing elements is completed, controlling the plurality of processing elements to shift the second non-zero elements input to the plurality of first processing elements to each of the plurality of second processing elements included in the second row, and When operations between non-zero elements are completed in the plurality of second processing elements, the plurality of processing elements are controlled to shift second non-zero elements shifted to the plurality of second processing elements to each of a plurality of third processing elements included in a third row among the plurality of processing elements.

7. The electronic device according to claim 6, wherein: The processor is further configured to, when a second non-zero element input to each of the plurality of processing elements belongs to the same row and the same column as a second non-zero element used in a previously executed operation, accumulate an operation result of the input second non-zero element with a previous operation result, and store the accumulated operation result in one of the plurality of register files.

8. The electronic device according to claim 6, wherein: The processor is further configured to: when a second non-zero element input to each of the plurality of processing elements does not belong to the same row and the same column as a second non-zero element used in a previously executed operation, shift an operation result stored in one of the plurality of register files of each of the plurality of processing elements to an adjacent processing element, accumulate the operation result of the input second non-zero element and the shifted operation result and store the accumulated operation result in one of the plurality of register files.

9. A method for controlling an electronic device to perform deep learning, wherein: The electronic device includes a processor, wherein the processor includes a plurality of processing elements arranged in a matrix shape, and the method includes: extracting only a plurality of first non-zero elements from a plurality of first elements included in target data stored in a memory, and inputting the extracted plurality of first non-zero elements to each of the plurality of processing elements, wherein the target data is three-dimensional data; extracting only a second non-zero element from a plurality of second elements included in the kernel data stored in the memory, and sequentially inputting the extracted second non-zero element to each of the plurality of first processing elements included in a first row among the plurality of processing elements, wherein the kernel data is three-dimensional data; and Only the operation between the second non-zero element input in each cycle and the first non-zero element of the plurality of first non-zero elements having the same depth information as the second non-zero element input is performed, The step of sequentially inputting the second non-zero elements comprises: when it is identified that there is no first non-zero element corresponding to a specific depth in all rows and columns of the first non-zero elements stored in each of the plurality of processing elements, omitting the input of the second non-zero elements corresponding to the specific depth in the plurality of second elements, and sequentially inputting the second non-zero elements not corresponding to the specific depth to each of the plurality of first processing elements, wherein the first element corresponding to the specific depth is a zero element, wherein each of the plurality of processing elements comprises a plurality of register files, and The first non-zero element of the input includes: Based on the row information and the column information of the first non-zero element, identifying a corresponding processing element from the plurality of processing elements, and The first non-zero element is sequentially input to the plurality of register files included in the identified processing element, and depth information of the first non-zero element is stored together with the first non-zero element.

10. The method of claim 9, wherein: The step of sequentially inputting the second non-zero element further includes sequentially inputting the second non-zero element to each of the plurality of first processing elements based on row information, column information, and depth information of the second non-zero element.

11. The method of claim 10, wherein: The step of sequentially inputting the second non-zero element also includes: sequentially inputting second non-zero elements included in one row and one column among the second non-zero elements to each of the plurality of first processing elements based on the depth; and When all second non-zero elements included in the one row and the one column are input to each of the plurality of first processing elements, second non-zero elements included in a row and a column different from the one row and the one column are input to each of the plurality of first processing elements.

12. The method of claim 11, wherein: The step of sequentially inputting the second non-zero element also includes: inputting zero to each of the plurality of first processing elements when there is no second non-zero element in the one row and the one column; and When zero is input to each of the plurality of first processing elements, second non-zero elements or zeros included in a row and a column different from the one row and the one column are input to each of the plurality of first processing elements based on the number of second non-zero elements included in the row and the column different from the one row and the one column.

Images