Electronic device and method for controlling electronic device
The electronic device optimizes quantized weight data by adjusting quantization levels based on input data properties, addressing the fixed weight data issue in neural networks and enhancing model precision and efficiency.
Patent Information
- Application Number
- PCT/KR2024/005614
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-04-25
- Publication Date
- 2024-12-05
AI Technical Summary
Existing neural network models face limitations in optimizing quantized weight data during inference processes, as the quantized weight data is fixed and not adapted to the properties of varying input data, leading to suboptimal precision and efficiency.
An electronic device and method that dynamically optimizes quantized weight data by obtaining latent vectors representing the properties of both the weight data and input data, using a quantization adjustment value acquisition module to adjust the quantization levels of the weights, thereby generating optimized second weight data for improved precision and efficiency.
The approach enhances the precision of neural network models by adaptively optimizing quantized weight data in real-time based on input properties, leading to improved accuracy and computational efficiency.
Smart Images

Figure KR2024005614_05122024_PF_FP_ABST
Abstract
Description
Electronic devices and methods of controlling electronic devices
[0001] The present disclosure relates to an electronic device and a method for controlling the electronic device, and more particularly, to an electronic device capable of optimizing quantized weight data of a neural network model to input data and a method for controlling the same.
[0002] Recently, technologies for making neural network models lighter have been developed, and these technologies are further developed as efforts continue to efficiently implement neural network models in on-device form.
[0003] In particular, in the technology of quantizing the weight data of a neural network model expressed in high-precision units into relatively low-precision weight data, maintaining the level of precision required by the neural network model while improving computational efficiency is emerging as a major challenge.
[0004] However, prior art techniques for quantization use weight data obtained through quantization as a fixed result, regardless of the input data, during the subsequent neural network model inference process. Therefore, limitations have been pointed out in these techniques, as the quantized weight data is not optimized to take into account the various attributes of the input data.
[0005] The present disclosure is intended to overcome the limitations of the prior art as described above, and an object of the present disclosure is to provide an electronic device and a control method thereof capable of improving the accuracy of a neural network model by optimizing quantized weight data of the neural network model to input data.
[0006] According to one embodiment of the present disclosure for achieving the above-described object, an electronic device includes a memory for storing quantized first weight data of a neural network model, and a processor for obtaining a first latent vector that compressively represents properties of the first weight data, obtaining a second latent vector that compressively represents properties of input data of the neural network model, obtaining a third latent vector by combining the first latent vector and the second latent vector, inputting the third latent vector to a quantization adjustment value obtaining module, obtaining a plurality of quantization adjustment values for changing a quantization level of each of a plurality of weights included in the first weight data, and obtaining second weight data in which the first weight data is changed such that it is optimized for the input data based on the first weight data and the plurality of quantization adjustment values.
[0007] Here, the processor can obtain the first latent vector by backpropagating a predefined loss used for learning the neural network model to the quantization adjustment value acquisition module.
[0008] Meanwhile, the processor inputs the input data into a latent vector acquisition module to acquire the second latent vector, and the latent vector acquisition module includes an encoder that encodes an input vector to acquire an encoding vector and a decoder that decodes the encoding vector to acquire an output vector, and the latent vector acquisition module can be trained to acquire the second latent vector based on a loss defined based on a difference between the input vector and the output vector.
[0009] Meanwhile, the processor can obtain the third latent vector by concatenating the first latent vector and the second latent vector or adding the first latent vector and the second latent vector.
[0010] Meanwhile, the quantization adjustment value acquisition module includes a hidden vector acquisition module and a vector size change module, and the hidden vector acquisition module acquires a plurality of hidden vectors corresponding to each of the plurality of layers based on the third latent vector, and the vector size change module includes layers corresponding to each of the plurality of layers, and can acquire the plurality of quantization adjustment values by changing the plurality of hidden vectors to correspond to the sizes of each of the plurality of layers.
[0011] Meanwhile, the processor can obtain the first weight data by performing quantization on the non-quantized weight data of the neural network model.
[0012] Meanwhile, the plurality of quantization adjustment values have a value of 0 or 1, and the processor can obtain a plurality of integer weights by adding the plurality of weights included in the first weight data and the plurality of quantization adjustment values corresponding to each of the plurality of weights, and obtain the second weight data by multiplying each of the plurality of integer weights by a step size of quantization.
[0013] According to one embodiment of the present disclosure for achieving the above-described object, a control method of an electronic device includes the steps of: obtaining a first latent vector that compressively represents properties of quantized first weight data of a neural network model; obtaining a second latent vector that compressively represents properties of input data of the neural network model; obtaining a third latent vector by combining the first latent vector and the second latent vector; inputting the third latent vector to a parameter obtaining module to obtain a plurality of quantization adjustment values for changing a quantization level of each of a plurality of weights included in the first weight data; and obtaining second weight data in which the first weight data is changed to be optimized for the input data based on the first weight data and the plurality of quantization adjustment values.
[0014] Here, the step of obtaining the first latent vector can obtain the first latent vector by backpropagating a predefined loss used for learning the neural network model to the quantization adjustment value obtaining module.
[0015] Meanwhile, the step of obtaining the second latent vector includes a step of inputting the input data into a latent vector obtaining module to obtain the second latent vector, and the latent vector obtaining module includes an encoder that encodes an input vector to obtain an encoding vector and a decoder that decodes the encoding vector to obtain an output vector, and the latent vector obtaining module can be trained to obtain the second latent vector based on a loss defined based on a difference between the input vector and the output vector.
[0016] Meanwhile, the step of obtaining the third latent vector can obtain the third latent vector by concatenating the first latent vector and the second latent vector or adding the first latent vector and the second latent vector.
[0017] Meanwhile, the quantization adjustment value acquisition module includes a hidden vector acquisition module and a vector size change module, and the hidden vector acquisition module acquires a plurality of hidden vectors corresponding to each of the plurality of layers based on the third latent vector, and the vector size change module includes layers corresponding to each of the plurality of layers, and can acquire the plurality of quantization adjustment values by changing the plurality of hidden vectors to correspond to the sizes of each of the plurality of layers.
[0018] Meanwhile, the control method of the electronic device can obtain the first weight data by performing quantization on the non-quantized weight data of the neural network model.
[0019] Meanwhile, the plurality of quantization adjustment values have a value of 0 or 1, and the step of obtaining the second weight data may obtain a plurality of integer weights by adding the plurality of weights included in the first weight data and the plurality of quantization adjustment values corresponding to each of the plurality of weights, and obtain the second weight data by multiplying each of the plurality of integer weights by a step size of quantization.
[0020] According to one embodiment of the present disclosure for achieving the above-described object, there is provided a non-transitory computer-readable recording medium including a program for executing a control method of an electronic device, the control method of the electronic device including the steps of: obtaining a first latent vector that compressively represents properties of quantized first weight data of a neural network model; obtaining a second latent vector that compressively represents properties of input data of the neural network model; obtaining a third latent vector by combining the first latent vector and the second latent vector; inputting the third latent vector to a parameter acquisition module to obtain a plurality of quantization adjustment values for changing a quantization level of each of a plurality of weights included in the first weight data; and obtaining second weight data in which the first weight data is changed to be optimized for the input data based on the first weight data and the plurality of quantization adjustment values.
[0021] FIG. 1 is a block diagram briefly illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure;
[0022] FIG. 2 is a block diagram illustrating a neural network model and a plurality of modules according to one or more embodiments of the present disclosure;
[0023] FIG. 3 is a block diagram detailing the configuration of a latent vector acquisition module according to one or more embodiments of the present disclosure;
[0024] FIG. 4 is a block diagram detailing the configuration of a quantization adjustment value acquisition module according to one or more embodiments of the present disclosure;
[0025] FIG. 5 is a block diagram detailing the configuration of an electronic device according to one or more embodiments of the present disclosure, and
[0026] FIG. 6 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.
[0027] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.
[0028] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.
[0029] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.
[0030] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.
[0031] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.
[0032] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.
[0033] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.
[0034] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).
[0035] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.
[0036] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.
[0037] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.
[0038] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.
[0039] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.
[0040] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.
[0041] FIG. 1 is a block diagram briefly illustrating a configuration of an electronic device (100) according to one or more embodiments of the present disclosure, and FIG. 2 is a block diagram illustrating a neural network model (1000) and a plurality of modules according to one or more embodiments of the present disclosure. In addition, FIG. 3 is a block diagram illustrating in detail a configuration of a latent vector acquisition module (2010) according to one or more embodiments of the present disclosure, and FIG. 4 is a block diagram illustrating in detail a configuration of a quantization adjustment value acquisition module (2030) according to one or more embodiments of the present disclosure. Hereinafter, descriptions will be made with reference to FIGS. 1 to 4 together.
[0042] The electronic device (100) according to the present disclosure refers to a device capable of optimizing quantized weight data of a neural network model (1000) to input data. For example, the electronic device (100) may be implemented as a server, an edge computing device, a personal computer (PC), a smartphone, etc., and there are no particular limitations on the type thereof.
[0043] As illustrated in FIG. 1, an electronic device (100) according to the present disclosure may include a memory (110) and a processor (120).
[0044] At least one instruction regarding the electronic device (100) may be stored in the memory (110). In addition, an O / S (Operating System) for driving the electronic device (100) may be stored in the memory (110). In addition, various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (110). In addition, the memory (110) may include a semiconductor memory (110) such as a flash memory (110) or a magnetic storage medium such as a hard disk.
[0045] Specifically, the memory (110) may store various software modules for operating the electronic device (100) according to various embodiments of the present disclosure, and the processor (120) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (110). That is, the memory (110) is accessed by the processor (120), and data reading / recording / modifying / deleting / updating, etc., may be performed by the processor (120).
[0046] Meanwhile, in the present disclosure, the term memory (110) may be used to mean a memory (110), a ROM (not shown), a RAM (not shown) in a processor (120), or a memory (110) card (not shown) (e.g., a micro SD card, a memory (110) stick) mounted on an electronic device (100).
[0047] In particular, in various embodiments according to the present disclosure, the memory (110) can store the quantized first weight data of the neural network model (1000). Specifically, the processor (120) can perform quantization on the non-quantized weight data of the neural network model (1000) to obtain the first weight data, and store the obtained first weight data in the memory (110). In addition, the quantization is performed by an external device, and the processor (120) can obtain the first weight data by receiving the first weight data from the external device through the communication unit (130), and store the obtained first weight data in the memory (110).
[0048] 'Quantization' refers to the process of converting weight data expressed in high-precision units into weight data of relatively low precision. In other words, quantization refers to the process of converting weight data expressed in the first bit range into weight data expressed in the second bit range, which is smaller than the first bit.
[0049] In the present disclosure, "unquantized weight data" may refer to weight data expressed in a 32-bit floating point (FP32) format. Furthermore, "first weight data" may refer to weight data having a bit width of less than 32 bits, i.e., weight data obtained as a result of quantization.
[0050] Meanwhile, the type of the neural network model (1000) according to the present disclosure and the type of weight data of the neural network model (1000) are not limited to a specific type, and there are no particular restrictions on the method of quantizing the weight data. For example, various embodiments according to the present disclosure can be applied to various quantization methods such as the single precision method, the mixed-precision method, the Quantization-Aware Training (QAT) method, and the Post-Training Quantization (PTQ) method.
[0051] Meanwhile, various data such as data for a neural network model (1000), data for various modules according to the present disclosure, latent vectors, multiple quantization adjustment values, data for learning the neural network model (1000) and / or various modules, etc. may be stored in the memory (110). In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (110), and the information stored in the memory (110) may be updated as received from an external device or input by a user.
[0052] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) is connected to the configuration of the electronic device (100) including the memory (110), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (110) as described above.
[0053] The processor (120) may be implemented in various ways. For example, the processor (120) may be implemented as at least one of an application specific integrated circuit (ASIC), an embedded processor (120), a microprocessor (120), hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (120) (Digital Signal Processor, DSP). Meanwhile, the term "processor (120)" in the present disclosure may be used to mean a central processing unit (CPU), a graphic processing unit (GPU), and a microprocessor unit (MPU).
[0054] In particular, in various embodiments according to the present disclosure, the processor (120) may optimize the quantized weight data (i.e., the first weight data as described above) of the neural network model (1000) to input data using a plurality of modules. At least one of the plurality of modules may be implemented as a software module or a hardware module.
[0055] Hereinafter, various embodiments implemented by a processor (120) and a plurality of modules according to the present disclosure will be described with reference to FIG. 2. As illustrated in FIG. 2, the plurality of modules may include a latent vector acquisition module (2010), a latent vector combination module (2020), a quantization adjustment value acquisition module (2030), and a weight data change module (2040).
[0056] Specifically, the processor (120) can obtain a first latent vector that compressively represents the properties of the first weight data. Here, the 'latent vector' refers to a low-dimensional vector that represents the core properties (or main features) of the given data, and can be said to be a vector that is compressively expressed as important properties / features of the given data are encoded (embedded). Meanwhile, in the present disclosure, the term 'vector' is not limited to one-dimensional data, and thus may be replaced with terms such as 'tensor' or 'matrix'.
[0057] In particular, in the present disclosure, the term "first latent vector" is used to refer to a vector that concisely represents the properties of the first weight data. The first latent vector can be obtained for each layer, each channel, or each block of the neural network model (1000).
[0058] Specifically, the processor (120) can train the quantization adjustment value acquisition module (2030) by backpropagating the loss used for learning the neural network model (1000) to the quantization adjustment value acquisition module (2030), and can obtain the first latent vector through the learned quantization adjustment value acquisition module (2030).
[0059] In other words, the loss used for training the neural network model (1000) can be backpropagated not only through multiple layers included in the neural network model (1000), but also through layers included in the quantization adjustment value acquisition module (2030). Accordingly, the quantization adjustment value acquisition module (2030) can be trained to acquire a first latent vector that compressively represents the properties of the first weight data. A more specific description of the quantization adjustment value acquisition module (2030) will be provided below.
[0060] Meanwhile, the loss used in the learning process of the quantization adjustment value acquisition module (2030) can be defined for each layer of the neural network model (1000) or each predefined block, and can also be defined to correspond to the entire process of the neural network model (1000). In addition, the type of loss can be defined as a function value of various loss functions such as L1 loss, L2 loss, MSE (mean squared error), and RMSE (root mean squared error), and is not limited to a specific type.
[0061] The processor (120) can obtain a second latent vector that compressively represents the properties of input data of the neural network model (1000). Specifically, the processor (120) can obtain the second latent vector using the latent vector obtaining module (2010). The latent vector obtaining module (2010) is a learned module including a neural network, and in particular, refers to a module that is learned to obtain the second latent vector according to the present disclosure. Here, the term 'second latent vector' refers to a term for a vector that compressively represents the properties of input data of the neural network model (1000).
[0062] As illustrated in FIG. 3, the latent vector acquisition module (2010) may be an auto-encoder including an encoder (2011) and a decoder (2012). The encoder (2011) may encode an input vector to obtain an encoding vector, and the decoder (2012) may decode the obtained encoding vector to obtain an output vector. In addition, the latent vector acquisition module (2010) may be trained to obtain a second latent vector based on a loss defined based on the difference between the input vector and the output vector.
[0063] In other words, the latent vector acquisition module (2010) can operate by encoding an input vector into an encoding vector and then decoding the latent vector to reconstruct a vector that is as similar as possible to the input vector. Accordingly, the encoding vector can compressively express the properties of the input vector, so when input data according to the present disclosure is input into the latent vector acquisition module (2010), a second latent vector that compressively expresses the properties of the input data can be acquired.
[0064] The processor (120) can obtain a third latent vector by combining the first latent vector and the second latent vector. Specifically, the processor (120) can obtain the third latent vector by using the latent vector combination module (2020). The latent vector combination module (2020) is a module that can obtain a third latent vector by combining the first latent vector and the second latent vector, and the 'third latent vector' can be defined as a vector that compressively represents the properties of the first weight data and the properties of the input data.
[0065] Specifically, the latent vector combination module (2020) can obtain a third latent vector by concatenating the first latent vector and the second latent vector or adding (i.e., adding) the first latent vector and the second latent vector. Here, the concatenation refers to an operation that connects vectors (or tensors) to create a large vector (or tensor) and expands the dimension. In addition to concatenation and addition, the latent vector combination module (2020) can perform various operations between the first latent vector and the second latent vector.
[0066] The processor (120) inputs the third latent vector into the quantization adjustment value acquisition module (2030), thereby obtaining a plurality of quantization adjustment values for changing the quantization level of each of the plurality of weights included in the first weight data. The quantization adjustment value acquisition module (2030) is a learned module including a neural network, and in particular refers to a module learned to obtain a quantization adjustment value according to the present disclosure.
[0067] In the present disclosure, the term "quantization adjustment value" is used as a general term for values for changing the quantization level of each of the plurality of weights included in the first weight data. Specifically, the plurality of quantization adjustment values are values for changing each of the plurality of weights included in the first weight data to be optimized for the input data, and can be obtained in the number of the plurality of weights.
[0068] In addition, the plurality of quantization adjustment values can be optimized between 0 and 1 in the learning process of the quantization adjustment value acquisition module (2030) as described above, and can ultimately converge to 0 or 1. That is, each of the plurality of quantization adjustment values has a value of 0 or 1, and thus can be used as a value for maintaining or increasing by 1 the quantization level of each of the plurality of weights included in the first weight data.
[0069] As illustrated in FIG. 4, the quantization adjustment value acquisition module (2030) may include a hidden vector acquisition module (2031) and a vector size change module (2032).
[0070] The hidden vector acquisition module (2031) refers to a module that can acquire a plurality of hidden vectors corresponding to each of a plurality of layers based on a third latent vector. Here, the plurality of hidden vectors refers to vectors for representing the properties of each of the plurality of layers. Specifically, the hidden vector refers to one of the activations of the layers included in the quantization adjustment value acquisition module (2030). Specifically, each of the plurality of hidden vectors can be said to be the third latent vector decomposed into each of the plurality of layers, and thus can correspond to the properties of each of the plurality of layers.
[0071] The vector size change module (2032) refers to a module that can obtain multiple quantization adjustment values by changing the size of a hidden vector. Specifically, the vector size change module (2032) includes layers corresponding to each of a plurality of layers, and can obtain multiple quantization adjustment values by changing a plurality of hidden vectors to correspond to the sizes of each of the plurality of layers.
[0072] Specifically, since the size of the weight data of each of the plurality of layers, i.e., the number of weights, may differ for each layer, the quantization adjustment value acquisition module (2030) needs to adjust the number of output values to correspond to the number of weights of each of the plurality of layers. Accordingly, the vector size change module (2032) can change the size of each of the plurality of hidden vectors corresponding to the properties of each of the plurality of layers. For example, the process of adjusting the number of output values to correspond to the number of weights of each of the plurality of layers may be performed according to a matrix multiplication operation for the hidden vectors corresponding to each of the plurality of layers and the weight data corresponding to each of the plurality of layers, but is not limited thereto.
[0073] The processor (120) can obtain second weight data, in which the first weight data is modified to be optimized for input data, based on the first weight data and a plurality of quantization adjustment values. Specifically, the processor (120) can obtain the second weight data using the weight data modification module (2040). The weight data modification module (2040) refers to a module that can change the first weight data into the second weight data.
[0074] The weight data change module (2040) can obtain multiple integer weights by adding up multiple weights included in the first weight data and multiple quantization adjustment values corresponding to each of the multiple weights. In addition, the weight data change module (2040) can finally obtain second weight data by multiplying each of the multiple integer weights by a quantization step size.
[0075] For example, if weight data expressed in the FP32 format is quantized into 8 bits to obtain first weight data expressed as an integer from 0 to 225, if the first weight included in the first weight data is 210 / 256 and the quantization adjustment value corresponding to the first weight is 1, the weight data change module (2040) can obtain a quantized weight of 211 / 256 by adding 1 to 201, which is an integer weight (quantization level) corresponding to the second weight, and then multiplying it by a step size of 1 / 256. Meanwhile, if the second weight included in the first weight data is 200 / 256 and the quantization adjustment value corresponding to the first weight is 0, the weight data change module (2040) can obtain a quantized weight of 201 / 256 by maintaining the integer weight corresponding to the second weight and multiplying it by a step size of 1 / 256.
[0076] Meanwhile, the first weight data and the second weight data described above can be expressed in mathematical formulas as follows.
[0077]
[0078]
[0079] Here, is the weight included in the first weight data, is the weight included in the second weight data, Is Integer weights of, is the quantization adjustment value, and represents the step size.
[0080] The processor (120) quantizes the weight data before quantization and the step size and integer weights Weight expressed as a product of can be obtained. And, according to the embodiments described above, the quantization adjustment value Once this is obtained, the processor (120) calculates the integer weight and quantization adjustment values Add up the sum and add the step size to the summed value The weights included in the second weight data can be obtained by multiplying them.
[0081] In embodiments according to the present disclosure, the update target is a quantization adjustment value. , and the quantization adjustment value can be obtained through learning in the latent space rather than the weight space. And, the quantization adjustment value Since the second latent vector reflects the properties of the input data, it is possible to obtain second weight data optimized according to the input data.
[0082] As described above, when the second weight data is acquired, the processor (120) can change the weights for each of the multiple layers of the neural network model (1000) into weights included in the second weight data, and perform an inference process on the input data to acquire output data. Specifically, the processor (120) can store the acquired second weight data in the memory (110), and perform an operation process on the input data based on the second weight data by loading the second weight data stored in the memory (110) prior to performing the inference process.
[0083] According to the various embodiments described above, the electronic device (100) can improve the accuracy of the neural network model (1000) by optimizing the quantized weight data of the neural network model to the input data. In other words, the electronic device (100) can generate weight data optimized in real time and dynamically by considering the properties of each of the various input data, thereby further improving the accuracy of the neural network model (1000).
[0084] Meanwhile, in the above, an embodiment of obtaining a third latent vector by combining the first latent vector and the second latent vector and inputting the third latent vector into the quantization adjustment value obtaining module (2030) to obtain multiple quantization adjustment values according to the present disclosure has been described. However, according to another embodiment, multiple quantization adjustment values may be obtained by inputting the first latent vector into the quantization adjustment value obtaining module (2030) without using the second latent vector.
[0085] That is, according to an embodiment of obtaining multiple quantization adjustment values by inputting only the first latent vector into the quantization adjustment value obtaining module (2030), since the second latent vector is not used, the latent vector obtaining module (2010) and the latent vector combining module (2020) of FIG. 2 are not used.
[0086] When only the first latent vector is input into the quantization adjustment value acquisition module (2030) to acquire multiple quantization adjustment values, since the second latent vector is not used, the properties of the input data are not reflected in the acquisition of the second weight data, but the properties of the quantized weight data can be reflected, and compared to the case where all parameters of the neural network model (1000) are used, the number of parameters used for learning can be relatively reduced, so there is an advantage in that optimization of the weight data can be achieved efficiently.
[0087] Meanwhile, the latent vector acquisition module (2010) and the quantization adjustment value acquisition module (2030) according to the present disclosure may include a neural network, and at least one of the latent vector acquisition module (2010) and the quantization adjustment value acquisition module (2030) may be implemented as a single integrated model with the neural network model (1000).
[0088] Meanwhile, the latent vector acquisition module (2010), the quantization adjustment value acquisition module (2030), and the neural network model (1000) can be individually trained based on losses defined for each module or model, and can also be comprehensively trained based on a single loss. For example, the latent vector acquisition module (2010) can be trained based on losses defined according to the difference between the input and output of each module, and can also be trained based on backpropagation of losses defined according to the difference between the input and output of the neural network model (1000).
[0089] Meanwhile, the above description assumes that the latent vector acquisition module (2010), the quantization adjustment value acquisition module (2030), and the neural network model (1000) are all implemented by the electronic device (100), but at least one of the latent vector acquisition module (2010), the quantization adjustment value acquisition module (2030), and the neural network model (1000) may be implemented by an external device, and the electronic device (100) may perform operations according to various embodiments of the present disclosure through data transmission and reception with the external device.
[0090] FIG. 5 is a block diagram detailing the configuration of an electronic device (100) according to one or more embodiments of the present disclosure.
[0091] As illustrated in FIG. 5, the electronic device (100) according to the present disclosure may further include a communication unit (130), an input unit (140), and an output unit (150) in addition to a memory (110) and a processor (120). However, the configurations illustrated in FIGS. 1 and 5 are merely exemplary, and it is to be understood that new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIGS. 1 and 5 when implementing the present disclosure.
[0092] The communication unit (130) includes a circuit and can perform communication with an external device. Specifically, the processor (120) can receive various data or information from an external device connected via the communication unit (130) and can also transmit various data or information to the external device.
[0093] The communication unit (130) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and a UWB (Ultra-Wide Band) module. Specifically, the WiFi module and the Bluetooth module may each perform communication in the WiFi or Bluetooth manner. When using a WiFi module or a Bluetooth module, various connection information, such as an SSID, may be first transmitted and received, and then communication may be established using this, after which various pieces of information may be transmitted and received.
[0094] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc. And, the NFC module can perform communication in the NFC (Near Field Communication) method using the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, 2.45GHz, etc. In addition, the UWB module can accurately measure ToA (Time of Arrival), which is the time it takes for a pulse to reach a target, and AoA (Ange of Arrival), which is the pulse arrival angle at the transmitting device, through communication between UWB antennas, and accordingly, precise distance and location recognition is possible within an error range of several tens of centimeters indoors.
[0095] In particular, in various embodiments according to the present disclosure, the processor (120) may receive various data, such as data for a neural network model (1000), data for various modules according to the present disclosure, latent vectors, multiple quantization adjustment values, data for learning the neural network model (1000) and / or various modules, from an external device through a communication unit (130).
[0096] In addition, when at least one of the latent vector acquisition module (2010), the quantization adjustment value acquisition module (2030), and the neural network model (1000) according to the present disclosure is implemented by an external device, the processor (120) may perform operations according to various embodiments according to the present disclosure by transmitting and receiving input / output data for the latent vector acquisition module (2010), the quantization adjustment value acquisition module (2030), and the neural network model (1000) from the external device through the communication unit (130).
[0097] The input unit (140) includes a circuit, and the processor (120) can receive a user command for controlling the operation of the electronic device (100) through the input unit (140). Specifically, the input unit (140) can include at least one of a microphone, a camera, and a remote control signal receiving unit. In addition, the input unit (140) can also be implemented in a form included in a display as a touch screen.
[0098] A camera can acquire an image of at least one object. Specifically, the camera includes an image sensor, and the image sensor can convert light entering through a lens into an electrical image signal.
[0099] The microphone can acquire a signal for a sound or voice generated from outside the electronic device (100). Specifically, the microphone can acquire a vibration corresponding to a sound or voice generated from outside the electronic device (100) and convert the acquired vibration into an electrical signal.
[0100] In particular, in various embodiments according to the present disclosure, the processor (120) may receive user inputs, such as user inputs for performing quantization on non-quantized weight data, a latent vector acquisition module (2010), a quantization adjustment value acquisition module (2030), and a user input for performing a learning process of a neural network model (1000), through an input unit (140).
[0101] The output unit (150) includes a circuit, and the processor (120) can output various functions that the electronic device (100) can perform through the output unit (150). In addition, the output unit (150) can include at least one of a display, a speaker, and an indicator.
[0102] The display can output image data under the control of the processor (120). Specifically, the display can output an image previously stored in the memory (110) under the control of the processor (120). In particular, the display according to an embodiment of the present disclosure can also display a user interface stored in the memory (110). The display can be implemented as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), etc., and in some cases, the display can also be implemented as a flexible display, a transparent display, etc. However, the display according to the present disclosure is not limited to a specific type.
[0103] The speaker can output audio data under the control of the processor (120), and the indicator can be turned on under the control of the processor (120).
[0104] In particular, in various embodiments according to the present disclosure, the processor (120) may output, through the output unit (150), information indicating that the quantization process according to the present disclosure has been completed, information indicating that the process of obtaining second weight data that has been changed to be optimized for input data has been completed, and the like.
[0105] FIG. 6 is a flowchart illustrating a method for controlling an electronic device (100) according to one or more embodiments of the present disclosure.
[0106] As illustrated in FIG. 6, the electronic device (100) can obtain a first latent vector that compressively represents the properties of the quantized first weight data of the neural network model (1000) (S610). Here, the first weight data refers to weight data obtained as a result of quantization of the weight data of the neural network model, and can be stored in the memory of the electronic device.
[0107] Specifically, the electronic device (100) can train the quantization adjustment value acquisition module (2030) by backpropagating the loss used for learning the neural network model (1000) to the quantization adjustment value acquisition module (2030), and can obtain the first latent vector through the learned quantization adjustment value acquisition module (2030).
[0108] The electronic device (100) can obtain a second latent vector that compressively represents the properties of input data of the neural network model (1000) (S620). Specifically, the electronic device (100) can obtain the second latent vector using a latent vector obtaining module (2010) that includes an encoder (2011) that encodes an input vector to obtain an encoding vector and a decoder (2012) that decodes the obtained encoding vector to obtain an output vector.
[0109] The electronic device (100) can obtain a third latent vector by combining the first latent vector and the second latent vector (S630). Specifically, the electronic device (100) can obtain the third latent vector by concatenating the first latent vector and the second latent vector or by adding the first latent vector and the second latent vector.
[0110] The electronic device (100) can input the third latent vector into the quantization adjustment value acquisition module (2030) to acquire a plurality of quantization adjustment values for changing the quantization level of each of the plurality of weights included in the first weight data (S640). Specifically, the electronic device (100) can acquire a plurality of hidden vectors corresponding to each of the plurality of layers based on the third latent vector, and acquire a plurality of quantization adjustment values by changing the size of the hidden vector.
[0111] The electronic device (100) can obtain second weight data in which the first weight data is modified to be optimized for input data based on the first weight data and the plurality of quantization adjustment values (S650). Specifically, the electronic device (100) can obtain a plurality of integer weights by adding the plurality of weights included in the first weight data and the plurality of quantization adjustment values corresponding to each of the plurality of weights. In addition, the weight data modification module (2040) can finally obtain the second weight data by multiplying each of the plurality of integer weights by the step size of quantization.
[0112] Meanwhile, the control method of the electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the control method of the electronic device (100) may be stored and provided in a non-transitory computer readable medium.
[0113] Specifically, in a non-transitory computer-readable recording medium including a program for executing a control method of an electronic device (100), the control method of the electronic device (100) may include a step of obtaining a first latent vector that compressively represents properties of quantized first weight data of a neural network model (1000), a step of obtaining a second latent vector that compressively represents properties of input data of the neural network model (1000), a step of obtaining a third latent vector by combining the first latent vector and the second latent vector, a step of inputting the third latent vector to a parameter obtaining module to obtain a plurality of quantization adjustment values for changing a quantization level of each of a plurality of weights included in the first weight data, and a step of obtaining second weight data in which the first weight data is changed to be optimized for the input data based on the first weight data and the plurality of quantization adjustment values.
[0114] In the above, a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100) have been briefly described, but this is only to omit redundant descriptions, and it goes without saying that various embodiments of the electronic device (100) can also be applied to a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100).
[0115] According to various embodiments of the present disclosure as described above, the electronic device (100) can improve the accuracy of the neural network model (1000) by optimizing the quantized weight data of the neural network model to the input data. In other words, the electronic device (100) can generate weight data optimized in real time and dynamically by considering the properties of each of various input data, thereby further improving the accuracy of the neural network model (1000).
[0116] The artificial intelligence-related function according to the present disclosure is operated through the processor (120) and memory (110) of the electronic device (100).
[0117] The processor (120) may be composed of one or more processors (120). At this time, the one or more processors (120) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Processing Unit), but is not limited to the examples of the processors (120) described above.
[0118] The CPU is a general-purpose processor (120) capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layer cache structure allows for the efficient execution of complex programs. The CPU is advantageous in a serial processing method, enabling organic linking of previous and subsequent calculation results through sequential calculations. The general-purpose processor (120) is not limited to the examples described above, except in cases where it is specifically referred to as a CPU.
[0119] A GPU is a processor (120) for large-scale operations such as floating point operations used in graphic processing, and can perform large-scale operations in parallel by integrating a large number of cores. In particular, a GPU may be advantageous compared to a CPU in parallel processing methods such as convolution operations. In addition, a GPU may be used as a co-processor (120) to supplement the functions of a CPU. The processor (120) for large-scale operations is not limited to the examples described above, except in cases where it is specified as a GPU as described above.
[0120] An NPU is a processor (120) specialized in artificial intelligence operations using an artificial neural network, and each layer constituting the artificial neural network can be implemented with hardware (e.g., silicon). At this time, since the NPU is designed specifically according to the required specifications of the company, it has a lower degree of freedom compared to a CPU or GPU, but it can efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor (120) specialized in artificial intelligence operations, the NPU can be implemented in various forms such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), a Vision Processing Unit (VPU), etc. The artificial intelligence processor (120) is not limited to the above-described examples, except in cases where it is specified as the above-described NPU.
[0121] Additionally, one or more processors (120) may be implemented as a SoC (System on Chip). In this case, the SoC may further include, in addition to one or more processors (120), a memory (110), and a network interface such as a bus for data communication between the processor (120) and the memory (110).
[0122] When a plurality of processors (120) are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the processors (120) among the plurality of processors (120). For example, the electronic device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors (120). However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor (120).
[0123] In addition, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in one processor (120). In particular, the electronic device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by utilizing multiple cores included in the processor (120).
[0124] One or more processors (120) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in the memory (110). The predefined operation rules or artificial intelligence models are characterized by being created through learning.
[0125] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI according to the present disclosure is implemented, or through a separate server / system.
[0126] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.
[0127] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.
[0128] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.
[0129] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smartphones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be at least temporarily stored or temporarily created in a device-readable storage medium, such as a memory (110) of a manufacturer's server, an application store's server, or an intermediary server.
[0130] Each of the components (e.g., modules or programs) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in the various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration.
[0131] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.
[0132] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).
[0133] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored in the storage medium and operating according to the called instructions.
[0134] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.
[0135] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.
Claims
1. In electronic devices, A memory for storing quantized first weight data of a neural network model; and Obtain a first latent vector that compressively represents the properties of the first weight data, Obtain a second latent vector that compressively represents the properties of the input data of the above neural network model, By combining the first latent vector and the second latent vector, a third latent vector is obtained, By inputting the third latent vector into a quantization adjustment value acquisition module, multiple quantization adjustment values are acquired for changing the quantization level of each of multiple weights included in the first weight data. An electronic device comprising: a processor for obtaining second weight data, the first weight data being changed to be optimized for the input data, based on the first weight data and the plurality of quantization adjustment values; 2. In paragraph 1, The above processor, By back propagating the loss used for learning the above neural network model to the above quantization adjustment value acquisition module, the above quantization adjustment value acquisition module is trained, An electronic device for obtaining the first latent vector through a learned quantization adjustment value obtaining module.
3. In paragraph 1, The above processor, By inputting the above input data into the latent vector acquisition module, the second latent vector is acquired, The above latent vector acquisition module includes an encoder that encodes an input vector to obtain an encoding vector and a decoder that decodes the encoding vector to obtain an output vector. An electronic device in which the latent vector acquisition module is trained to acquire the second latent vector based on a loss defined based on a difference between the input vector and the output vector.
4. In paragraph 1, The above processor, An electronic device that obtains the third latent vector by concatenating the first latent vector and the second latent vector or by adding the first latent vector and the second latent vector.
5. In paragraph 1, The above quantization adjustment value acquisition module includes a hidden vector acquisition module and a vector size change module, The above hidden vector acquisition module acquires a plurality of hidden vectors corresponding to each of the plurality of layers based on the third latent vector, An electronic device in which the above vector size changing module includes layers corresponding to each of the plurality of layers, and obtains the plurality of quantization adjustment values by changing the plurality of hidden vectors to correspond to the sizes of each of the plurality of layers.
6. In paragraph 1, The above processor, An electronic device that obtains the first weight data by performing quantization on the non-quantized weight data of the neural network model.
7. In paragraph 1, The above multiple quantization adjustment values have values of 0 or 1, The above processor, A plurality of integer weights are obtained by adding a plurality of weights included in the first weight data and a plurality of quantization adjustment values corresponding to each of the plurality of weights, An electronic device that obtains the second weight data by multiplying each of the plurality of integer weights by a step size of quantization.
8. In a method for controlling an electronic device, A step of obtaining a first latent vector that compressively represents the properties of the quantized first weight data of the neural network model; A step of obtaining a second latent vector that compressively represents the properties of input data of the above neural network model; A step of obtaining a third latent vector by combining the first latent vector and the second latent vector; A step of inputting the third latent vector into a parameter acquisition module to acquire a plurality of quantization adjustment values for changing the quantization level of each of the plurality of weights included in the first weight data; and A method for controlling an electronic device, comprising: a step of obtaining second weight data in which the first weight data is changed to be optimized for the input data based on the first weight data and the plurality of quantization adjustment values; 9. In paragraph 8, The step of obtaining the first potential vector is: A control method of an electronic device for obtaining the first latent vector by back propagating a predefined loss used for learning the neural network model to the quantization adjustment value obtaining module.
10. In paragraph 8, The step of obtaining the second potential vector is: A step of inputting the above input data into a latent vector acquisition module to acquire the second latent vector; including; The above latent vector acquisition module includes an encoder that encodes an input vector to obtain an encoding vector and a decoder that decodes the encoding vector to obtain an output vector. A control method of an electronic device in which the latent vector acquisition module is trained to acquire the second latent vector based on a loss defined based on a difference between the input vector and the output vector.
11. In paragraph 8, The step of obtaining the third potential vector is: A control method of an electronic device, wherein the third potential vector is obtained by concatenating the first potential vector and the second potential vector or by adding the first potential vector and the second potential vector.
12. In paragraph 8, The above quantization adjustment value acquisition module includes a hidden vector acquisition module and a vector size change module, The above hidden vector acquisition module acquires a plurality of hidden vectors corresponding to each of the plurality of layers based on the third latent vector, A control method of an electronic device, wherein the vector size changing module includes layers corresponding to each of the plurality of layers, and obtains the plurality of quantization adjustment values by changing the plurality of hidden vectors to correspond to the sizes of each of the plurality of layers.
13. In paragraph 8, The method of controlling the above electronic device is: A control method of an electronic device for obtaining the first weight data by performing quantization on non-quantized weight data of the neural network model.
14. In paragraph 8, The above multiple quantization adjustment values have values of 0 or 1, The step of obtaining the second weight data is: A plurality of integer weights are obtained by adding a plurality of weights included in the first weight data and a plurality of quantization adjustment values corresponding to each of the plurality of weights, A control method of an electronic device for obtaining the second weight data by multiplying each of the plurality of integer weights by a step size of quantization.
15. A non-transitory computer-readable recording medium including a program for executing a method of controlling an electronic device, The method of controlling the above electronic device is: A step of obtaining a first latent vector that compressively represents the properties of the quantized first weight data of the neural network model; A step of obtaining a second latent vector that compressively represents the properties of input data of the above neural network model; A step of obtaining a third latent vector by combining the first latent vector and the second latent vector; A step of inputting the third latent vector into a parameter acquisition module to acquire a plurality of quantization adjustment values for changing the quantization level of each of the plurality of weights included in the first weight data; A computer-readable recording medium comprising: a step of obtaining second weight data in which the first weight data is changed to be optimized for the input data based on the first weight data and the plurality of quantization adjustment values;
Citation Information
Patent Citations
Document type recommendation method and apparatus, electronic device and readable storage medium
KR1020210038473A
Compressed recurrent neural network model
KR102100977B1
Apparatus and method for encoding / decoding audio signal using information of previous frame
US20210166706A1
Streamlined development and deployment of autoencoders
WO2022212215A1
Quantization range estimation for quantized training
WO2022221885A2
