Electronic apparatus and method for controlling thereof

Abstract

An electronic device for performing operations of a neural network model is disclosed. The electronic device includes a memory in which weight data including quantized weight values ​​of a neural network model is stored, and a processor that acquires operation data based on binary data and input data in which at least one bit value is different, generates a lookup table in which the operation data is matched to the binary data, acquires operation data corresponding to the weight data from the lookup table, and performs operations of the neural network model based on the acquired operation data.

Description

Technology Field

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same that operate based on artificial intelligence technology. Background Technology

[0002] Recently, artificial intelligence systems capable of achieving human-level intelligence are being developed. Unlike conventional rule-based systems, artificial intelligence systems are systems in which machines learn and make decisions on their own, and they are being utilized in a wide range of fields such as speech recognition, image recognition, and future prediction.

[0003] In particular, recently, artificial intelligence systems that solve given problems through deep neural networks based on deep learning are being developed.

[0004] A deep neural network is a neural network that includes multiple hidden layers between an input layer and an output layer, and refers to a model that implements artificial intelligence technology through neurons included in each layer.

[0005] Deep neural networks like this generally include a large number of neurons to produce accurate results.

[0006] However, when a vast number of neurons exist, aside from the increased accuracy of the output value relative to the input value, there is a problem in that the computation required to derive the output value consumes a significant amount of time.

[0007] In addition, due to the vast amount of neurons, deep neural networks cannot be used on mobile devices such as smartphones with limited memory due to capacity issues. The problem to be solved

[0008] The present disclosure is devised to solve the aforementioned problems, and the purpose of the present disclosure is to provide an electronic device that accurately derives an output value for an input value within a short period of time and enables the realization of artificial intelligence technology even on mobile devices with limited memory. means of solving the problem

[0009] An electronic device for performing operations of a neural network model according to one embodiment of the present disclosure includes a memory in which weight data including quantized weight values ​​of the neural network model is stored, and a processor that acquires operation data based on binary data and input data in which at least one bit value is different, generates a lookup table in which the operation data is matched to the binary data, acquires operation data corresponding to the weight data from the lookup table, and performs operations of the neural network model based on the acquired operation data.

[0010] Here, each of the above binary data consists of n bit values, and the input data includes a plurality of input values ​​of a matrix, and the processor obtains n input values ​​from each column of the matrix and can obtain the operation data for each of the binary data based on the binary data and the n input values.

[0011] And, the weight data includes a plurality of weight values ​​of a matrix, and the processor identifies n weight values ​​corresponding to n input values ​​in each row of the matrix, identifies binary data corresponding to the identified n weight values ​​among the binary data, obtains operation data corresponding to the identified binary data from the lookup table, and can perform operations of the neural network model based on the obtained operation data.

[0012] And, the processor can determine, among a plurality of lookup tables generated based on the input values ​​of each column of the matrix, a lookup table corresponding to each column of the output matrix for the input data, and obtain the output values ​​of each column of the output matrix from each of the lookup tables.

[0013] And, the processor can divide a matrix containing the plurality of input values ​​into a first matrix and a second matrix based on a preset row, divide a matrix containing the plurality of weight values ​​into a third matrix and a fourth matrix based on a preset column, generate a plurality of lookup tables based on the input values ​​of each column of the first matrix, obtain operation data corresponding to each row of the third matrix from the plurality of lookup tables, generate a plurality of lookup tables based on the input values ​​of each column of the second matrix, and obtain operation data corresponding to each row of the fourth matrix from the plurality of lookup tables.

[0014] And, the processor can obtain eight input values ​​from each column of the matrix and obtain the operation data for each binary data based on the binary data and the eight input values.

[0015] In addition, the processor may, in a plurality of operation expressions based on the binary data and the n input values, if there is a first operation expression and a second operation expression having the same intermediate operation expression, perform the operation of the second operation expression based on the operation value of the first operation expression.

[0016] A control method for an electronic device that performs operations of a neural network model according to one embodiment of the present disclosure comprises the steps of: obtaining operation data based on binary data and input data in which at least one bit value is different; generating a lookup table in which the operation data is matched to the binary data; obtaining operation data corresponding to weight data including quantized weight values ​​of the neural network model from the lookup table; and performing operations of the neural network model based on the obtained operation data.

[0017] Here, each of the above binary data consists of n bit values, and the input data includes a plurality of input values ​​of a matrix, and the step of obtaining the operation data involves obtaining n input values ​​from each column of the matrix and obtaining the operation data for each of the binary data based on the binary data and the n input values.

[0018] And, the weight data includes a plurality of weight values ​​of a matrix, and the step of performing the operation of the neural network model may identify n weight values ​​corresponding to n input values ​​in each row of the matrix, identify binary data corresponding to the identified n weight values ​​among the binary data, obtain operation data corresponding to the identified binary data from the lookup table, and perform the operation of the neural network model based on the obtained operation data.

[0019] And, the step of performing the operation of the neural network model may include determining, among a plurality of lookup tables generated based on the input values ​​of each column of the matrix, a lookup table corresponding to each column of the output matrix for the input data, and obtaining the output values ​​of each column of the output matrix from each of the lookup tables.

[0020] And, the step of obtaining the above-mentioned operation data may involve dividing a matrix containing the plurality of input values ​​into a first matrix and a second matrix based on a preset row, dividing a matrix containing the plurality of weight values ​​into a third matrix and a fourth matrix based on a preset column, generating a plurality of lookup tables based on the input values ​​of each column of the first matrix, obtaining operation data corresponding to each row of the third matrix from the plurality of lookup tables, generating a plurality of lookup tables based on the input values ​​of each column of the second matrix, and obtaining operation data corresponding to each row of the fourth matrix from the plurality of lookup tables.

[0021] And, the step of obtaining the above-mentioned operation data may involve obtaining eight input values ​​from each column of the above-mentioned matrix, and obtaining the above-mentioned operation data for each of the above-mentioned binary data based on the above-mentioned binary data and the above-mentioned eight input values.

[0022] And, the step of generating the lookup table may include, in the case where there are a first expression and a second expression having the same intermediate expression among a plurality of expressions based on the binary data and the n input values, the operation of the second expression is performed based on the operation value of the first expression. Effects of the invention

[0023] According to various embodiments of the present disclosure as described above, an output value for an input value can be accurately derived within a short period of time, and artificial intelligence technology can be realized even on mobile devices with limited memory. Brief explanation of the drawing

[0024] FIG. 1 is a block diagram for illustrating an electronic device according to one embodiment of the present disclosure. FIG. 2a is a diagram illustrating a matrix corresponding to input data according to one embodiment of the present disclosure. FIG. 2b is a drawing illustrating a lookup table according to one embodiment of the present disclosure. FIG. 3a is a diagram illustrating the operation of a neural network model using a lookup table according to one embodiment of the present disclosure. FIG. 3b is a drawing for explaining a lookup table used for each column of a matrix corresponding to output data according to one embodiment of the present disclosure. FIG. 3c is a drawing for explaining an embodiment of obtaining the output value of the first column of a matrix corresponding to the output data according to one embodiment of the present disclosure. FIG. 3d is a drawing for explaining an embodiment of obtaining the output value of the second column of a matrix corresponding to the output data according to one embodiment of the present disclosure. FIG. 3e is a drawing for explaining an embodiment of obtaining the output value of the third column of a matrix corresponding to the output data according to one embodiment of the present disclosure. FIG. 4a is a drawing for explaining an operation formula used to generate a lookup table according to one embodiment of the present disclosure. FIG. 4b is a drawing for explaining the same intermediate operation formula included in a plurality of operation formulas according to one embodiment of the present disclosure. FIG. 4c is a drawing for explaining the same intermediate operation formula included in a plurality of operation formulas according to one embodiment of the present disclosure. FIG. 4d is a drawing for explaining the same intermediate operation formula included in a plurality of operation formulas according to one embodiment of the present disclosure. FIG. 4e is a drawing for explaining the same intermediate operation formula included in a plurality of operation formulas according to one embodiment of the present disclosure. FIG. 4f is a drawing for explaining an embodiment of obtaining an operation value based on the same intermediate operation formula according to one embodiment of the present disclosure. FIG. 5 is a diagram illustrating a method of computation for a neural network model according to one embodiment of the present disclosure. FIG. 6 is a detailed block diagram for explaining an electronic device according to one embodiment of the present disclosure. FIG. 7 is a flowchart for explaining a method of controlling an electronic device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0025] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings. However, this is not intended to limit the technology described in the present disclosure to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives to the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0026] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0027] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0028] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0029] When it is stated that a component (e.g., a first component) is "(operatively or communicatively) coupled with" or "connected to" another component (e.g., a second component), it should be understood that the component may be directly connected to the other component or connected through another component (e.g., a third component). On the other hand, when it is stated that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between the component and the other component.

[0030] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some contexts, the expression “device configured to” may mean that the device is “capable of” in conjunction with other devices or components. For example, the phrase “processor configured to perform A, B, and C” may mean a dedicated processor for performing the said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in a memory device.

[0031] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor (not shown), except for a "module" or "part" that needs to be implemented in specific hardware.

[0032] The present disclosure will be described in detail below with reference to the attached drawings.

[0033] FIG. 1 is a block diagram for illustrating an electronic device according to one embodiment of the present disclosure.

[0034] Referring to FIG. 1, an electronic device (100) according to one embodiment of the present disclosure includes a memory (110) and a processor (120).

[0035] An electronic device (100) according to one embodiment of the present disclosure is a device that obtains output data for input data using a neural network model (or an artificial intelligence model). For example, the electronic device (100) may be a desktop PC, a laptop, a smartphone, a tablet PC, a server, etc. Alternatively, the electronic device (100) may be the system itself in which a cloud computing environment is established. However, it is not limited thereto, and the electronic device (100) may be any device capable of performing computations of a neural network model.

[0036] The memory (110) can be implemented as a hard disk, non-volatile memory, or volatile memory. Here, the non-volatile memory can be an OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, etc., and the volatile memory can be a DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc.

[0037] Meanwhile, in FIG. 1, the memory (110) is shown as being separate from the processor (120), but the memory (110) may be included in the processor (120). That is, the memory (110) may be implemented as an off-chip memory, as well as as an on-chip memory.

[0038] Additionally, although one memory (110) is shown in FIG. 1, the memory (110) may be implemented in multiple numbers according to the embodiment.

[0039] The memory (110) can store weight data of a neural network model. Here, the weight data is data used for computation of the neural network model, and the memory (110) can store multiple weight data corresponding to multiple layers constituting the neural network model.

[0040] In particular, the memory (110) can store weight data including quantized weight values. Here, the quantized weight value can be -1 or 1, and the weight data can be represented as an m×n matrix composed of -1 or 1. Additionally, the weight value -1 can be replaced with 0 and stored in the memory (110). That is, the memory (110) can store weight data composed of 0 or 1. According to an embodiment, weight data including weight values ​​of -1 or 1 can be stored in a first memory (e.g., a hard disk), and weight data including weight values ​​of 0 or 1 can be stored in a second memory (e.g., an SDRAM). Here, the weight values ​​of 0 or 1 can be used in the operation of the neural network model to be described later.

[0041] Meanwhile, the quantization of the neural network model can be performed by the processor (120) of the electronic device (100), as well as by an external device (e.g., a server). When the quantization of the neural network model is performed by an external device, the processor (120) can receive weight data containing quantized weight values ​​from the external device and store it in memory (110). The method for quantizing the neural network model will be described later.

[0042] Such a neural network model can be a model based on a neural network. For example, a neural network model can be a model based on a Recurrent Neural Network (RNN). Here, RNN stands for Recurrent Neural Network and is a type of deep learning model designed to learn data that changes over time, such as time-series data.

[0043] However, it is not limited to this, and the neural network model can be a model based on various networks such as CNN (Convolutional Neural Network), DNN (Deep Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), or BRDNN (Bidirectional Recurrent Deep Neural Network). Alternatively, the memory (110) may store a model generated based on rules rather than a model learned through an artificial intelligence algorithm, and there are no special restrictions on the model stored in the memory (110).

[0044] The processor (120) controls the overall operation of the electronic device (100). To this end, the processor (120) may be composed of one or more processors. Here, the one or more processors may be general-purpose processors such as a central processing unit (CPU), as well as graphics-dedicated processors such as a graphics processing unit (GPU) or artificial intelligence-dedicated processors such as a neural network processing unit (NPU). Additionally, the processor (120) may be a System on Chip (SoC) (e.g., an On-device AI Chip), a Large Scale Integration (LSI), or a Field Programmable Gate Array (FPGA).

[0045] The processor (120) can quantize the weight values ​​of a neural network model. Specifically, the processor (120) can quantize the weight values ​​of a neural network model through various quantization algorithms that satisfy the following mathematical formula 1 when quantizing the weight values ​​into k bits.

[0046] [Mathematical Formula 1]

[0047]

[0048] (Here, w is the weight value before quantization, a is the scaling factor, and b is the quantized weight value, which can be -1 or +1.)

[0049] For example, the processor (120) can quantize weight values ​​through a greedy algorithm. In this case, the processor (120) can obtain the scaling factor and the quantized weight values ​​when k=1 in the above-described mathematical formula 1 based on the following mathematical formula 2.

[0050] [Mathematical Formula 2]

[0051]

[0052] (Here, w is the weight value before quantization, a* is the scaling factor for k=1, b* is the quantized weight value for k=1, which is -1 or +1, and n can be an integer greater than or equal to 1.)

[0053] And, the processor (120) can obtain the scaling factor and quantized weight value for the case where k = i (1 < i ≤ k) by repeatedly calculating the following mathematical formula 3. That is, the processor (120) can obtain the scaling factor and quantized weight value for the case where k = i (1 < i ≤ k) by using r, which is the difference between the weight value before quantization and the quantized weight value when k = 1.

[0054] [Mathematical Formula 3]

[0055]

[0056] (Here, w is the weight value before quantization, a is the scaling factor, b is the quantized weight value, which is -1 or +1, and r can be the difference between the weight value before quantization and the quantized weight value when k=1.)

[0057] Accordingly, the electronic device (100) can store weight data including a scaling factor and weight values ​​quantized to -1 or 1 in memory (110). Meanwhile, although an example of quantization using a greedy algorithm has been described above, there are no particular limitations on the method of quantizing weight values. For example, quantization can be performed using various algorithms such as unitary quantization, adaptive quantization, uniform quantization, or supervised iterative quantization.

[0058] The processor (120) can obtain output data for input data based on the quantized weight values ​​of the neural network model. Here, the input data may be text, an image, or user voice, etc. For example, the text may be text entered through an input unit (not shown), such as a keyboard or touchpad of the electronic device (100), and the image may be an image captured through the camera of the electronic device (100). Additionally, the user voice may be user voice input into the microphone of the electronic device (100).

[0059] Meanwhile, the output data may vary depending on the type of input data and / or neural network model. That is, the output data may vary depending on which input data is input into which neural network model. For example, if the neural network model of the present disclosure is a model for language translation, the processor (120) may obtain output data expressed in a second language for input data expressed in a first language. Alternatively, if the neural network model of the present disclosure is a model for image analysis, the processor (120) may input an image as input data for the neural network model and obtain information regarding an object detected in the image as output data. Furthermore, if the neural network model of the present disclosure is a model for speech recognition, the processor (120) may obtain user voice as input data and text corresponding to the user voice as output data. Meanwhile, the above-described output data is merely an example, and the types of output data of the present disclosure are not limited thereto.

[0060] To this end, when input data is input, the processor (120) can represent the input data as a matrix (or vector or tensor) containing multiple input values. Here, the method of representing the input data as a matrix (or vector or tensor) may vary depending on the type and category of the input data. For example, when text (or text converted from user voice) is input as input data, the processor (120) can represent the text as a vector through one-hot encoding or through word embedding. Here, one-hot encoding is a method of representing only the index value of a specific word as 1 and the values ​​of the remaining indices as 0, and word embedding is a method of representing words as real numbers using a vector dimension (e.g., 128 dimensions) set by the user. Examples of word embedding methods may include Word2Vec, FastText, Glove, etc. Meanwhile, if the processor (120) receives an image as input data, it can represent each pixel of the image as a matrix. For example, the processor (120) can represent each pixel of the image as a value from 0 to 255 for each RGB color, or represent the image as a matrix by dividing the value from 0 to 255 by a preset value (e.g., 255).

[0061] The processor (120) can obtain at least one intermediate data for the input data and at least one output data for the intermediate data based on the quantized weight values ​​and the input values ​​of the input data.

[0062] Here, unlike conventional electronic devices that obtain output data for input data by performing matmul operations on a plurality of quantized weight values ​​and a plurality of input values, the present disclosure can obtain output data for input data using a lookup table.

[0063] This is to prevent latency issues and memory overload phenomena that occur during multiple matmul operations. First, the lookup table of the present disclosure will be described in detail with reference to FIGS. 2a and 2b.

[0064] FIGS. 2a and 2b are drawings for illustrating a lookup table according to one embodiment of the present disclosure.

[0065] As described above, the processor (120) can acquire input data in the form of a matrix (or a vector or tensor) containing multiple input values. For example, the processor (120) can acquire a 4×3 matrix containing multiple input values ​​as shown in FIG. 2a. Hereinafter, a matrix containing multiple input values ​​is referred to as an input matrix (or a matrix corresponding to input data).

[0066] The processor (120) can generate a lookup table based on the input values ​​of the input data and the binary data. Here, the binary data may be data composed of n bit values ​​having a value of 0 or 1. And, the number of binary data may be 2^n. For example, 2 bits of binary data may be data composed of 2 bit values ​​having a value of 0 or 1, and may be one of 00, 01, 10, or 11. For another example, 4 bits of binary data may be data composed of 4 bit values ​​having a value of 0 or 1, and may be one of 0000, 0001, 0010, 0011, 0100, 0101, 0110, 0111, 1000, 1001, 1010, 1011, 1100, 1101, 1110, or 1111.

[0067] Specifically, the processor (120) can obtain n input values ​​from each column of the input matrix and obtain operation data based on the bit values ​​of the binary data and the obtained n input values. Here, the bit value of the binary data can be 0 or 1 as described above, and the processor (120) can perform -1 operation on the input value when the bit value of the binary data is 0, and perform 1 operation on the input value when the bit value of the binary data is 1. Then, the processor (120) can generate a lookup table by matching the operation data obtained for each binary data.

[0068] For example, referring to FIG. 2, when n = 2, the processor (120) can obtain input values ​​of 0.03 and -0.17 of the first and second rows in the first column of the input matrix. Then, the processor (120) can obtain an operation value of 0.14 by performing -(0.03)-(-0.17) and match the operation value of 0.14 to binary data 00. Similarly, the processor (120) can obtain an operation value of -0.20 by performing -(0.03)+(-0.17) and match the operation value of -0.20 to binary data 01, obtain an operation value of 0.20 by performing (0.03)-(-0.17) and match the operation value of 0.20 to binary data 10, obtain an operation value of -0.14 by performing (0.03)+(-0.17) and match the operation value of -0.14 to binary data 11. Also, the processor (120) can obtain input values ​​of 0.20 and -0.17 for the third and fourth rows in the first column. And, the processor (120) can calculate -(0.20)-(0.17) to obtain a calculated value of -0.37 and match the calculated value of -0.37 to binary data 00. Similarly, the processor (120) can calculate -(0.20)+(0.17) to obtain a calculated value of -0.03 and match the calculated value of -0.03 to binary data 01, calculate (0.20)-(0.17) to obtain a calculated value of 0.03 and match the calculated value of 0.03 to binary data 10, and calculate (0.20)+(0.17) to obtain a calculated value of 0.37 and match the calculated value of 0.37 to binary data 11. In a similar manner, the processor (120) can obtain operation values ​​for the input values ​​of the second column and the input values ​​of the third column of the input matrix and match the operation values ​​to binary data. Meanwhile, the above-described case where n = 2 is an example, and n can be changed according to user settings.

[0069] Accordingly, the processor (120) can generate at least one lookup table for each column of the input matrix. As described above, if the input matrix is ​​a 4×3 matrix and n = 2, the processor (120) can generate two lookup tables for each column of the input matrix, i.e., a total of six lookup tables, as shown in FIG. 2b. If n = 4, the processor (120) can generate one lookup table for each column of the input matrix, i.e., a total of three lookup tables.

[0070] FIGS. 3a to 3e are drawings for explaining the operation of a neural network model using a lookup table according to one embodiment of the present disclosure.

[0071] As described above, the processor (120) can obtain output data based on quantized weight values ​​and input values ​​of input data. Specifically, the processor (120) can obtain output data for input data X based on the operation of weight data W (which may include a scaling factor A and a quantized weight value B) and input data X. For example, if the weight values ​​of the weight data are quantized into 3 bits, the processor (120) can obtain output data for input data X based on Equation 4 as follows.

[0072] [Mathematical Formula 4]

[0073]

[0074] Mathematical formula 4 can be expressed in matrix form as follows.

[0075]

[0076] For example, when k=1, the scaling factor Ao, the quantized weight Bo, and the input data X may have values ​​such as those shown in FIG. 3a. Here, unlike conventional electronic devices that obtain the operation value of B*X through matmul operations, the present disclosure can obtain the operation value of B*X through a lookup table reference. For convenience of explanation, a method for obtaining the operation value of Bo*X is described below, but the following technical concept can be considered applicable to cases where the operation value of Bn*X, such as B1*X, B2*X, etc., is obtained.

[0077] Referring to FIG. 3b, an output matrix containing the operation values ​​of Bo*X may have output values ​​y1 to y109. Here, the processor (120) may determine a lookup table corresponding to each column of the output matrix among a plurality of lookup tables generated for each column of the input matrix to obtain output values. Specifically, the processor (120) may determine a lookup table generated based on an input matrix with the same column as the output matrix as a lookup table for obtaining output values ​​of the output matrix. For example, referring to FIG. 3b, the processor (120) may determine a first lookup table (311) and a second lookup table (312) generated based on the input values ​​of the first column of the input matrix among a plurality of lookup tables as lookup tables for obtaining output values ​​of the first column of the output matrix.

[0078] And, the processor (120) can identify n weight values ​​corresponding to n input values ​​in each row of a weight matrix containing weight values ​​of 0 or 1. That is, if a lookup table is created by obtaining n input values ​​from each column of an input matrix, the processor (120) can identify n weight values ​​corresponding to n input values ​​in each row of a weight matrix. For example, as described above, when n=2, the processor (120) can identify two weight values ​​corresponding to two input values ​​for each row in a matrix containing weight values ​​(hereinafter referred to as the weight matrix or the matrix corresponding to weight data). That is, in FIG. 3c, the processor (120) can identify 1 and 0 and 0 and 1 in the first row of the weight matrix, identify 0 and 1 and 1 and 0 in the second row, and similarly identify two weight values ​​in the remaining rows.

[0079] And, the processor (120) can identify binary data corresponding to the identified weight values ​​among the binary data. Here, the identified binary data is data containing bit values ​​identical to the weight values, for example, as shown in FIG. 3c, if the identified weight values ​​are 1 and 0, the binary data corresponding to the weight values ​​can be 10, and if the identified weight values ​​are 0 and 1, the binary data corresponding to the weight values ​​can be 10.

[0080] And, the processor (120) can obtain an operation value corresponding to the binary data identified from the lookup table.

[0081] To this end, the processor (120) can determine among a plurality of lookup tables (e.g., the first and second lookup tables (311, 312) described above) a lookup table that includes operation values ​​corresponding to identified weight values. Specifically, if the identified weight values ​​are values ​​included in the k column of the weight matrix, the processor (120) can determine among the plurality of lookup tables a lookup table generated based on the input values ​​of the k row of the input matrix as a lookup table that includes operation values ​​corresponding to identified weight values. For example, referring to FIG. 3c, the processor (120) may determine, among the first and second lookup tables (311, 312), that the first lookup table (311), generated based on the input values ​​of the first and second rows of the input matrix, is a lookup table containing operation values ​​corresponding to the identified weight values ​​if the identified weight values ​​are values ​​included in the first and second columns of the weight matrix, and among the first and second lookup tables (311, 312), that the second lookup table (312), generated based on the input values ​​of the fourth and fourth rows of the input matrix, is a lookup table containing operation values ​​corresponding to the identified weight values ​​if the identified weight values ​​are values ​​included in the third and fourth columns of the weight matrix.

[0082] And, the processor (120) can obtain an operation value of 0.20 that matches binary data 10 identified from the first lookup table (311), obtain an operation value of -0.37 that matches binary data 01 identified from the second lookup table (312), and obtain the y1 value of the output matrix through the sum of the operation values ​​0.20 and 0.37. Similarly, for the second row of the weight matrix, the processor (120) can obtain an operation value of -0.20 that matches binary data 01 identified from the first lookup table (311), obtain an operation value of 0.37 that matches binary data 10 identified from the second lookup table (312), obtain the y4 value of the output matrix through the sum of the operation values ​​-0.20 and 0.37, and for the nth row of the weight matrix, obtain the output value in a similar manner.

[0083] Additionally, the output values ​​of the second column of the output matrix can be obtained from the third and fourth lookup tables (313, 314) as shown in FIG. 3d, and the output values ​​of the third column of the output matrix can be obtained from the fifth and sixth lookup tables (315, 316) as shown in FIG. 3e. Here, since the technical concept described above can be applied, a detailed explanation is omitted.

[0084] Subsequently, when the output values ​​of the output matrix are obtained, the processor (120) performs operations on the output matrix and the scaling factor, and accordingly obtains the result value of the mathematical formula 4 described above. Then, the processor (120) can output final output data using the obtained result value. Here, as described above, if the neural network model is a model for language translation, the output data may be text in a language different from the text input as input data, and if the neural network model is a model for image analysis, the output data may be data containing information about objects included in the image, but is not necessarily limited thereto.

[0085] As such, the present disclosure obtains output values ​​through a lookup table without performing matmul operations on quantized weight values ​​and input values ​​of input data, thereby preventing latency problems and memory overload phenomena caused by a massive amount of computation.

[0086] FIGS. 4a to 4f are drawings for explaining an operation formula used to generate a lookup table according to one embodiment of the present disclosure.

[0087] As described above, the processor (120) can obtain operation values ​​through a plurality of operation expressions based on binary data and n input values, and generate a lookup table by matching operation values ​​for each binary data.

[0088] Here, multiple operation expressions based on binary data and n input values ​​may include the same intermediate operation expression. For example, when n=8, multiple operation expressions based on 8-bit binary data and 8 input values ​​(x0 to x7) may be as shown in FIG. 4a. That is, when n=8, the operation expression for obtaining the operation value Ro corresponding to the binary data 00000000 is -x0-x1-x2-x3-x4-x5-x6-x7, and the operation expression for obtaining the operation value R1 corresponding to the binary data 00000001 is -x0-x1-x2-x3-x4-x5-x6+x7, and similarly, there may be multiple operation expressions for obtaining multiple operation values ​​corresponding to each of R2 to R255. In this case, the processor (120) may use the result value of an intermediate operation expression that is identically included in the multiple operation expressions when obtaining the operation values ​​based on the multiple operation expressions.

[0089] Specifically, referring to FIG. 4b, the operation formula -x0-x1-x2-x3-x4-x5-x6-x7 for obtaining Ro and the operation formula -x0-x1-x2-x3-x4-x5-x6-x7 for obtaining R1 include the same intermediate operation formula -x0-x1-x2-x3-x4-x5-x6. And, referring to FIG. 4c, the operation formula -x0-x1-x2-x3-x4-x5-x6-x7 for obtaining Ro and the operation formula -x0-x1-x2-x3-x4-x5+x6-x7 for obtaining R2 may include the same intermediate operation formula -x0-x1-x2-x3-x4-x5-x7, and the operation formula -x0-x1-x2-x3-x4-x5-x6-x7 for obtaining R1 and the operation formula -x0-x1-x2-x3-x4-x5+x6+x7 for obtaining R3 may include the same intermediate operation formula x0-x1-x2-x3-x4-x5+x7. Additionally, as shown in FIG. 4d and FIG. 4e, there may be multiple operation formulas for obtaining operation values ​​that include the same intermediate operation formula.

[0090] In this case, the processor (120) can perform the operation of one of the multiple operation expressions having the same intermediate operation expression based on the operation value of another operation expression. That is, as in the embodiment described above, if the operation expression -x0-x1-x2-x3-x4-x5-x6-x7 for obtaining Ro and the operation expression -x0-x1-x2-x3-x4-x5-x6-x7 for obtaining R1 include the same intermediate operation expression -x0-x1-x2-x3-x4-x5-x6, the processor (120) can obtain Ro through the operation expression -x0-x1-x2-x3-x4-x5-x6-x7 and obtain R1 by adding 2*x7 to the operation value of R0 (i.e., R1=R0+2x7). In a similar manner, the processor (120) can obtain values ​​from R2 to R255, and consequently, the processor (120) can obtain operation values ​​through a plurality of operation expressions shown in FIG. 4f. Meanwhile, although the case where n is 8 has been described as an example here, it should be understood that the technical concept described above can be applied as is even when n is a different integer.

[0091] In this way, if the same intermediate expression is included, the operation of one expression is performed using the operation value of another expression, thereby the present disclosure has the effect of significantly reducing the amount of computation of a processor for generating a lookup table.

[0092] FIG. 5 is a diagram illustrating a method of computation for a neural network model according to one embodiment of the present disclosure.

[0093] As described above, the processor (120) can generate a lookup table and perform operations of a neural network model based on the lookup table. Here, the processor (120) can generate multiple lookup tables for all input values ​​of the input data and perform operations of a neural network model based on the multiple lookup tables, and can also generate some lookup tables for some input values ​​of the input data, perform some operations of a neural network model based on the some lookup tables, generate some lookup tables for the remaining input values ​​of the input data, and perform the remaining operations of a neural network model based on the some lookup tables.

[0094] Specifically, the processor (120) may divide the input matrix into a first input matrix and a second input matrix based on a preset row, and divide the weight matrix into a third weight matrix and a fourth weight matrix based on a preset column. Here, the preset row may be n / 2 rows if the number of rows in the input matrix is ​​n, and the preset column may be n / 2 columns if the number of columns in the weight matrix is ​​n, but is not necessarily limited thereto.

[0095] And, the processor (120) may generate multiple lookup tables based on the input values ​​of each column of the first input matrix and obtain operation data corresponding to each row of the third weight matrix from the multiple lookup tables, and then generate multiple lookup tables based on the input values ​​of each column of the second input matrix and obtain operation data corresponding to each row of the fourth weight matrix from the multiple lookup tables. Here, the method of generating the lookup tables and obtaining operation data based on the lookup tables may be applied using the technical concept described above, so specific omissions are omitted.

[0096] For example, referring to FIG. 5, when the number of rows of the input matrix and the number of columns of the weight matrix are each 512, the processor (120) can divide the input matrix into an input matrix X1 and an input matrix X2 based on 256 rows, and divide the weight matrix into a weight matrix W1 and a weight matrix W2 based on 256 columns. Then, the processor (120) can generate a lookup table based on the input values ​​of the input matrix X1, obtain operation values ​​corresponding to the weight matrix W1 from the lookup table, generate a lookup table based on the input values ​​of the input matrix X2, and obtain operation values ​​corresponding to the weight matrix W2 from the lookup table. By obtaining multiple operation values ​​in part through the divided matrices in this way, the present disclosure can prevent memory overhead problems and utilize memory efficiently. In addition, if the processor (120) of the present disclosure is implemented in multiple units according to an embodiment, the time required for computation can be reduced by obtaining computation values ​​corresponding to weight matrix W1 from a lookup table based on input matrix X1 and computation values ​​corresponding to weight matrix W2 from a lookup table based on input matrix X2 in parallel through multiple processors.

[0097] FIG. 6 is a detailed block diagram for explaining an electronic device according to one embodiment of the present disclosure.

[0098] Referring to FIG. 6, according to one embodiment of the present disclosure, a first memory (110), a processor (120), a lookup table generator (LUT Generator) (130), a second memory (140), and a multiplier (150) may be included.

[0099] The first memory (110) can store an input matrix, a scaling factor for computation of a neural network model, and a weight matrix. Here, the input matrix may include a plurality of input values ​​as described above, and the weight matrix may include weight values ​​quantized to 0 or 1.

[0100] The lookup table generator (130) can load an input matrix from the first memory (110). The lookup table generator (130) can obtain operation values ​​for the input values ​​of the input matrix for each binary data. Specifically, when the lookup table generator (130) generates a lookup table for n bits of binary data, it can obtain n input values ​​from each column of the input matrix and obtain operation values ​​for each binary data based on the binary data and the n input values. The lookup table generator (130) can store information about the columns and rows of the input matrix that formed the basis for generating the lookup table by matching them in the lookup table. Here, the information about the columns can be used when determining the lookup table corresponding to each column of the output matrix among a plurality of lookup tables generated for each column of the input matrix. And, information about the row can be used when determining a lookup table containing an operation value corresponding to each column of the weight matrix among a plurality of lookup tables corresponding to each column of the output matrix.

[0101] Meanwhile, according to an embodiment, the lookup table generator (130) can generate a lookup table based on 8-bit binary data. Specifically, the lookup table generator (130) can obtain 8 input values ​​from each column of an input matrix and obtain operation data for each binary data based on the 8-bit binary data and the 8 input values. This takes into account that a processor (120), such as a CPU, processes data in byte units. Accordingly, the present disclosure can prevent overload of the processor by not separately performing shift operations, etc., to process bit-level data into byte-unit data.

[0102] The second memory (140) can store at least one lookup table. Here, the second memory (140) may be a scratchpad memory (SPM) that temporarily stores data such as a lookup table, but is not necessarily limited thereto.

[0103] The processor (120) can load a weight matrix from the first memory (110) and load a lookup table from the second memory (140). Then, the processor (120) can obtain the operation values ​​of the weight values ​​of the weight matrix from the lookup table, accumulate them in an accumulator, and obtain the output values ​​of the output matrix (i.e., weight matrix B * input matrix X) based on the sum of the operation values ​​accumulated in the accumulator. Then, the processor (120) can store information about the output values ​​of the output matrix in the first memory (110). Afterwards, the multiplier (150) can load the output values ​​and scaling factor stored in the first memory (110) and perform a multiplication operation of the output values ​​and scaling factor.

[0104] Meanwhile, although the first memory (110) and the second memory (120) are depicted as separate configurations in FIG. 6, the first memory (110) and the second memory (120) may be a single memory. Additionally, the first memory (110) and the second memory (120) may be separate configurations from the processor (120), but may also be included within the processor (120). Furthermore, the lookup table generator (130) may be implemented as a separate operation unit for generating a lookup table, and the function of the lookup table generator may be installed in the processor (120). Additionally, the multiplier (150) may be implemented as a separate operation unit that multiplies a scaling factor to an operation value obtained from the lookup table, and the function of the multiplier (150) may be installed in the processor (120).

[0105] Meanwhile, the above description explains an example of generating a lookup table based on the input values ​​of input data. However, an electronic device (100) according to one embodiment of the present disclosure may generate a lookup table based on the weight values ​​of weight data by applying the above-described method in reverse. That is, the processor (120) may quantize the input values ​​of input data while keeping the weight values ​​as they are (i.e., processing them as real values). Then, the processor (120) may generate a lookup table in which operation values ​​are matched for each binary data based on the weight values ​​and n-bit binary data, and obtain operation data corresponding to the input data from the lookup table. Such a lookup table based on weight data may be used for the operation of a language model in which the size of the input data is small and the size of the weight data is large. Meanwhile, the lookup table based on input data described above in FIG. 2, etc. may be used for the operation of an image model in which the size of the input data is large and the size of the weight data is small.

[0106] FIG. 7 is a drawing for explaining a method of controlling an electronic device according to one embodiment of the present disclosure.

[0107] The electronic device (100) can acquire operation data (S710) based on binary data and input data in which at least one bit value is different. Here, the binary data may be data composed of bit value 0 or bit value 1, the input data may be data containing multiple input values, and the operation data may be data containing multiple operation values. In addition, each of the input data and operation data may be represented as a matrix. Specifically, the electronic device (100) can acquire n input values ​​from each column of the input matrix and acquire operation data for each binary data based on the binary data and the n input values.

[0108] And, the electronic device (100) can generate a lookup table in which operation data is matched to binary data (S720) and obtain operation data corresponding to weight data from the lookup table (S730). Here, the weight data may include multiple weight values ​​of a matrix. Specifically, the electronic device (100) can identify n weight values ​​corresponding to n input values ​​in each row of the weight matrix and identify binary data corresponding to the identified n weight values ​​among the binary data. And, the electronic device (100) can obtain operation data corresponding to the identified binary data from the lookup table and perform an operation of a neural network model based on the obtained operation data (S740).

[0109] Meanwhile, the methods according to the various embodiments of the present disclosure described above may be implemented in the form of software or applications that can be installed on existing electronic devices.

[0110] In addition, a non-transitory computer-readable medium may be provided, which stores a program for sequentially performing a control method of an electronic device according to the present disclosure.

[0111] A non-transient readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on non-transient readable media such as CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0112] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0113] 100: Electronic device 110: Memory 120: Processor

Claims

Claim 1 An electronic device for performing operations on a neural network model, comprising: a memory in which a weight matrix including a plurality of quantized weight values ​​of the neural network model is stored; and a processor that acquires n input values ​​from each column of an input matrix including a plurality of input values, acquires a plurality of binary data each including n bit values, acquires operation data for each of the plurality of binary data based on the plurality of binary data and the n input values, generates a lookup table in which the operation data is matched for each of the plurality of binary data, identifies n weight values ​​corresponding to the n input values ​​in each row of the weight matrix, identifies binary data among the plurality of binary data that corresponds to the identified n weight values, acquires operation data corresponding to the identified binary data from the lookup table, and performs operations on the neural network model based on the operation data corresponding to the identified binary data. Claim 2 delete Claim 3 delete Claim 4 An electronic device according to claim 1, wherein the processor determines, among a plurality of lookup tables generated based on the n input values ​​of each column of the input matrix, a lookup table corresponding to each column of the output matrix for the n input values, and obtains output values ​​of each column of the output matrix from each of the lookup tables. Claim 5 An electronic device according to claim 1, wherein the processor divides the input matrix into a first matrix and a second matrix based on a preset row, divides the weight matrix into a third matrix and a fourth matrix based on a preset column, generates a plurality of lookup tables based on input values ​​of each column of the first matrix, obtains operation data corresponding to each row of the third matrix from the plurality of lookup tables, generates a plurality of lookup tables based on input values ​​of each column of the second matrix, and obtains operation data corresponding to each row of the fourth matrix from the plurality of lookup tables. Claim 6 An electronic device according to claim 1, wherein the processor obtains eight input values ​​from each column of the input matrix and obtains operation data for each of the plurality of binary data based on the plurality of binary data and the eight input values. Claim 7 An electronic device according to claim 1, wherein, in a plurality of operation expressions based on the plurality of binary data and the n input values, if there are a first operation expression and a second operation expression having the same intermediate operation expression, the operation of the second operation expression is performed based on the operation value of the first operation expression. Claim 8 A method for controlling an electronic device that performs operations of a neural network model, comprising: a step of obtaining n input values ​​from each column of an input matrix containing a plurality of input values; a step of obtaining a plurality of binary data each containing n bit values; a step of obtaining operation data for each of the plurality of binary data based on the plurality of binary data and the n input values; a step of generating a lookup table in which the operation data is matched for each of the plurality of binary data; a step of identifying n weight values ​​corresponding to the n input values ​​in each row of a weight matrix containing a plurality of quantized weight values; a step of identifying binary data among the plurality of binary data that corresponds to the identified n weight values; a step of obtaining operation data corresponding to the identified binary data from the lookup table; and a step of performing operations of the neural network model based on the operation data corresponding to the identified binary data. Claim 9 delete Claim 10 delete Claim 11 A method for controlling an electronic device according to claim 8, wherein the step of performing operations of the neural network model comprises determining, among a plurality of lookup tables generated based on input values ​​of each column of the input matrix, a lookup table corresponding to each column of the output matrix for the n input values, and obtaining output values ​​of each column of the output matrix from each of the lookup tables. Claim 12 In claim 8, the step of obtaining operation data corresponding to the identified binary data comprises dividing the input matrix into a first matrix and a second matrix based on a preset row, dividing the weight matrix into a third matrix and a fourth matrix based on a preset column, generating a plurality of lookup tables based on input values ​​of each column of the first matrix, obtaining operation data corresponding to each row of the third matrix from the plurality of lookup tables, generating a plurality of lookup tables based on input values ​​of each column of the second matrix, and obtaining operation data corresponding to each row of the fourth matrix from the plurality of lookup tables. Claim 13 In claim 8, the step of obtaining operation data for each of the plurality of binary data comprises obtaining eight input values ​​from each column of the input matrix and obtaining operation data for each of the plurality of binary data based on the plurality of binary data and the eight input values. Claim 14 A method for controlling an electronic device according to claim 8, wherein the step of generating the lookup table comprises, in the case where there are a first expression and a second expression having the same intermediate expression among a plurality of expressions based on the plurality of binary data and the n input values, the operation of the second expression is performed based on the operation value of the first expression.

Images