Method and apparatus for evaluating quantized artificial neural network

Abstract

A method and an apparatus for evaluating a quantized artificial neural network are disclosed. The method for evaluating a quantized artificial neural network, according to one embodiment, comprises the steps of: generating one or more original feature maps for input data by using a first artificial neural network model; determining the importance of each element of the one or more original feature maps; generating one or more quantized feature maps for the input data by using a second artificial neural network model, which is a quantized artificial neural network model for the first artificial neural network model; and calculating an evaluation value for the second artificial neural network model on the basis of the one or more original feature maps, the one or more quantized feature maps, and the importance.

Description

Quantized artificial neural network evaluation method and device

[0001] The disclosed embodiments relate to artificial intelligence technology.

[0002] Recently, in the field of artificial intelligence, quantization technology has been attracting attention as one of the methods for reducing the computational load of artificial neural network models and improving power efficiency.

[0003] Meanwhile, a method has been proposed to evaluate the performance of a quantized model by using a validation dataset and a method to evaluate the quantized model by using the difference (e.g., MSE (Mean Square Error)) between the feature map generated from the original model and the feature map generated from the quantized model. However, the former method requires securing a separate dataset for validation and labeling the secured dataset, and it is often difficult to secure a dataset for validation due to privacy issues. In the latter method, a smaller MSE is assumed to indicate better performance of the quantized model, but there is a problem that the correlation between MSE and the performance of the quantized model is weak.

[0004] The disclosed embodiments are intended to provide a method and apparatus for evaluating a quantized artificial neural network.

[0005] A quantized artificial neural network evaluation method according to one embodiment includes the steps of: generating one or more original feature maps for input data using a first artificial neural network model; determining importance for each element of the one or more original feature maps; generating one or more quantized feature maps for the input data using a second artificial neural network model, which is a quantized artificial neural network model for the first artificial neural network model; and calculating an evaluation value for the second artificial neural network model based on the one or more original feature maps, the one or more quantized feature maps, and the importance.

[0006] The one or more original feature maps may include feature maps generated from one or more layers among a plurality of layers included in the first artificial neural network model, and the one or more quantized feature maps may include quantized feature maps generated from layers corresponding to each layer of the first artificial neural network model that generated the one or more original feature maps among a plurality of layers included in the second artificial neural network model.

[0007] The above quantized artificial neural network evaluation method further includes the steps of generating one or more transformed data for the input data; and the step of generating one or more feature maps for each of the one or more transformed data using the first artificial neural network model, wherein the step of determining the importance may determine the importance for each element of the one or more original feature maps based on the one or more original feature maps and the one or more feature maps for each of the one or more transformed data.

[0008] The step of determining the importance may determine the importance based on the difference between corresponding elements among each element of each of the one or more original feature maps and each element of each of the one or more transformed data.

[0009] The step of determining the above importance can determine the above importance using a metric learning loss function.

[0010] The step of determining the importance may determine the importance using the gradient of the output of the first artificial neural network model for the input data of each of the one or more original feature maps.

[0011] The step of calculating the evaluation value may calculate the evaluation value based on the distance between each of the one or more original feature maps considering the importance and a feature map corresponding to each of the one or more original feature maps among the one or more quantized feature maps.

[0012] The step of calculating the evaluation value may calculate the evaluation value based on a distance between a result of applying the importance of each element of a feature map generated in an ith layer of the first artificial neural network model among the one or more original feature maps to a feature map generated in an ith layer of the first artificial neural network model and a result of applying the importance of each element of a feature map generated in an ith layer of the first artificial neural network model to a feature map generated in an ith layer of the second artificial neural network model among the one or more quantized feature maps.

[0013] A quantized artificial neural network evaluation device according to one embodiment comprises: one or more processors; and a memory storing one or more programs executed by the one or more processors, wherein the one or more processors generate one or more original feature maps for input data using a first artificial neural network model, determine importance for each element of the one or more original feature maps, generate one or more quantized feature maps for the input data using a second artificial neural network model which is a quantized artificial neural network model for the first artificial neural network model, and calculate an evaluation value for the second artificial neural network model based on the one or more original feature maps, the one or more quantized feature maps, and the importance.

[0014] The one or more original feature maps may include feature maps generated from one or more layers among a plurality of layers included in the first artificial neural network model, and the one or more quantized feature maps may include quantized feature maps generated from layers corresponding to each layer of the first artificial neural network model that generated the one or more original feature maps among a plurality of layers included in the second artificial neural network model.

[0015] The one or more processors may generate one or more transformed data for the input data, generate one or more feature maps for each of the one or more transformed data using the first artificial neural network model, and determine an importance for each element of the one or more original feature maps based on the one or more original feature maps and the one or more feature maps for each of the one or more transformed data.

[0016] The one or more processors may determine the importance based on a difference between corresponding elements of each element of the one or more original feature maps and each element of the one or more transformed data.

[0017] The one or more processors may determine the importance using a metric learning loss function.

[0018] The one or more processors may determine the importance using a gradient of an output of the first artificial neural network model for each of the one or more original feature maps.

[0019] The one or more processors may calculate the evaluation value based on the distance between each of the one or more original feature maps considering the importance and a feature map corresponding to each of the one or more original feature maps among the one or more quantized feature maps.

[0020] The one or more processors may calculate the evaluation value based on a distance between a result of applying the importance of each element of a feature map generated in an ith layer of the first artificial neural network model among the one or more original feature maps to a feature map generated in an ith layer of the first artificial neural network model and a result of applying the importance of each element of a feature map generated in an ith layer of the first artificial neural network model to a feature map generated in an ith layer of the second artificial neural network model among the one or more quantized feature maps.

[0021] According to the disclosed embodiments, a separate verification data set is not required to verify a quantized model or a labeling task for the data set is not required, and the importance of each element of a feature map generated by an original model before quantization is used to evaluate the performance of the quantized model, thereby enabling a more precise evaluation of the quantized model compared to the prior art.

[0022] Figure 1 is a schematic diagram of an evaluation device according to one embodiment.

[0023] FIG. 2 is a flowchart of a quantized artificial neural network evaluation method according to one embodiment.

[0024] FIG. 3 is a block diagram illustrating a computing environment including a computing device according to one embodiment.

[0025] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the present invention is not limited thereto.

[0026] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing embodiments of the present invention and should not be limited in any way. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0027] Figure 1 is a schematic diagram of an evaluation device according to one embodiment.

[0028] Referring to FIG. 1, an evaluation device (100) according to one embodiment includes a feature map generation unit (110), an importance calculation unit (120), and an evaluation unit (130).

[0029] In one embodiment, the feature map generation unit (110), the importance calculation unit (120), and the evaluation unit (130) may be implemented using one or more physically separate devices, or may be implemented by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operations, unlike the illustrated example.

[0030] The feature map generation unit (110) generates one or more original feature maps for input data using a first artificial neural network model, and generates one or more quantized feature maps for the input data using a second artificial neural network model, which is a quantized artificial neural network model for the first artificial neural network model.

[0031] In one embodiment, the input data may be multidimensional matrix data, such as an image, for example.

[0032] In one embodiment, the first artificial neural network model may be a deep neural network model including a plurality of layers configured to generate a feature map for input data, such as, for example, a convolutional neural network (CNN). However, the first artificial neural network model is not necessarily limited thereto and may be configured as an artificial neural network of various structures including a plurality of layers configured to generate a feature map for an input image.

[0033] Meanwhile, according to one embodiment, the second artificial neural network model may be a model generated by quantizing the output of the weights and activation functions of the artificial neural network constituting the first artificial neural network model. Specifically, the second artificial neural network model may be a model generated by converting the data format of the output of the weights and activation functions of the first artificial neural network model into a data format expressed with a smaller number of bits through quantization while maintaining the neural network structure of the first artificial neural network model.

[0034] For example, if the outputs of the weights and activation functions of the first artificial neural network model are in a fixed-point data format expressed in 32 bits, the second artificial neural network model may be a model that converts the outputs of the weights and activation functions of the first artificial neural network model into a fixed-point data format expressed in 16 bits or an integer data format expressed in 8 bits using a preset mapping function. However, the data formats of the first artificial neural network model and the second artificial neural network model and the quantization method used to generate the second artificial neural network model are not necessarily limited to a specific example.

[0035] According to one embodiment, the one or more original feature maps for the input data may include feature maps generated in one or more layers among a plurality of layers included in the first artificial neural network model. Furthermore, the one or more quantized feature maps for the input data may include feature maps generated in layers corresponding to each layer of the first artificial neural network model that generated the one or more original feature maps for the input data among a plurality of layers included in the second artificial neural network model. Specifically, the one or more original feature maps for the input data may include feature maps generated in an i-th layer of the first artificial neural network model (where i is a positive integer 1 ≤ i ≤ n, and n is the number of layers included in the first artificial neural network model), and the one or more quantized feature maps for the input data may include feature maps generated in an i-th layer of the second artificial neural network model.

[0036] Meanwhile, according to one embodiment, the feature map generation unit (110) may generate one or more transformed data for input data, and generate one or more feature maps for each of the one or more transformed data using the first artificial neural network model. At this time, the transformed data may be generated, for example, by adding a perturbation, such as random noise, to the input data. In addition, the one or more feature maps for the transformed data may include feature maps generated for the transformed data in each layer that generated one or more original feature maps for the input data among the multiple layers of the first artificial neural network model.

[0037] The importance calculation unit (120) determines the importance for each element of one or more original feature maps for input data.

[0038] At this time, the importance of each element of the original feature map may indicate the sensitivity of each element to changes in the input data. For example, the importance calculation unit (120) may determine the importance of each element of the original feature map so that elements with relatively large changes in the input data have higher importance, and elements with relatively small changes have lower importance.

[0039] According to one embodiment, the importance calculation unit (120) may determine the importance of each element of one or more original feature maps for the input data based on one or more original feature maps for the input data and one or more feature maps for each of one or more transformed data. Specifically, the importance calculation unit (120) may determine the importance of each element of one or more original feature maps for the input data based on a difference between corresponding elements among each element of one or more original feature maps for the input data and each element of one or more feature maps for each of one or more transformed data.

[0040] For example, the difference between each element of a feature map generated from the ith layer of the first artificial neural network model among one or more original feature maps for input data and each element of a feature map generated from the ith layer of the first artificial neural network model for the jth transformed data among one or more transformed data can be calculated according to the following mathematical expression 1.

[0041] [Mathematical Formula 1]

[0042]

[0043] In Equation 1, x is the input data, ε j is the perturbation added to the input data x to generate the j-th transformed data, F (i) (x) is the feature map generated by the i-th layer of the first artificial neural network model for the input data, F (i) (x+ε j ) represents the feature map generated by the ith layer of the first artificial neural network model for the jth transformed data. In addition, d1 is F (i) Each element of (x) and F (i) (x+ε j ) represents a function for calculating the difference between corresponding elements among each element of D. j (i) (x) is F (i) Each element of (x) and F (i) (x+ε j ) represents a difference matrix containing the difference values ​​between corresponding elements among each element of F. Specifically, F (i) (x), F (i) (x+ε j ) and D j (i) (x) is a multidimensional matrix of the same size, D j (i) Each element of (x) is F (i) (x) and F (i) (x+ε j ) represents the difference between the corresponding elements in each. For example, F (i) (x), F(i) (x+ε j ) and D j (i) If (x) are two-dimensional matrices of the same size, D j (i) The first element of the first row of (x) is F (i) The first element of the first row of (x) and F (i) (x+ε j ) can represent the difference between the first element of the first row.

[0044] Meanwhile, the feature map F generated by the i-th layer of the first artificial neural network model among one or more original feature maps for the input data (i) The importance for each element of (x) can be calculated, for example, according to the mathematical expression 2 below.

[0045] [Equation 2]

[0046]

[0047] In mathematical expression 2, m is the number of transformed data, S is each difference matrix D j (i) F using the corresponding elements of (x) (i) A function that calculates the importance for each element of (x), I (i) is F (i) Represents an importance matrix containing importance values ​​for each element of (x). Here, S may be a function that calculates, for example, the p-norm, sample variance, etc. for the corresponding elements of each difference matrix.

[0048] Specifically, D j (i) (x), F (i) (x) and I (i) can be a multidimensional matrix of the same size. Also, I (i) Each element of F (i) (x) can represent the importance of the corresponding element, and each difference matrix D j (i)(x) can be derived through the p-norm or sample variance for the corresponding elements. For example, D j (i) (x), F (i) (x) and I (i) If I are two-dimensional matrices of the same size, (i) The first element of the first row of F (i) (x) represents the importance of the first element of the first row of each difference matrix D j (i) It can be derived through the p-norm or sample variance for the first elements of the first row of (x).

[0049] Meanwhile, according to another embodiment, the importance calculation unit (120) may determine the importance of each element of one or more original feature maps for input data using a metric learning loss function. At this time, the metric learning loss function may be, for example, contrastive loss, triplet loss, margin loss, N-pair loss, etc.

[0050] As a concrete example, if the metric learning loss function is a contrastive loss function, the feature map F generated by the i-th layer of the first artificial neural network model among one or more original feature maps for the input data (i) The importance for each element of (x) can be calculated according to mathematical expression 3 or mathematical expression 4 below.

[0051] [Equation 3]

[0052]

[0053] [Equation 4]

[0054]

[0055] As another example, if the metric learning loss function is a triplet loss function, the feature map F (i)The importance of each element of (x) can be calculated according to the mathematical expression 5 below.

[0056] [Equation 5]

[0057]

[0058] Also, as another example, if the metric learning loss function is an N-pair loss function, the feature map F (i) The importance of each element of (x) can be calculated according to the mathematical expression 6 below.

[0059] [Equation 6]

[0060]

[0061] Meanwhile, in mathematical equations 3 to 6, l metric represents the metric learning loss function. Also, x - is a negative sample generated from x, x + represents a positive sample generated from x. In this case, a negative sample may mean a sample whose difference from x is relatively large compared to a positive sample, and a positive sample may mean a sample whose difference from x is relatively small compared to a negative sample.

[0062] According to another embodiment, the importance calculation unit (120) may determine the importance of each element of one or more feature maps for the input data by using the gradient of each of one or more original feature maps for the input data. For example, a feature map F generated by the i-th layer of the first artificial neural network model among one or more original feature maps for the input data (i) The importance of each element of (x) can be calculated according to the mathematical formula 7 below.

[0063] [Equation 7]

[0064]

[0065] In mathematical expression 7, O(x) represents the target output of the first artificial neural network model for the input data or the difference between the output of the first artificial neural network model for the input data and the target output.

[0066] The evaluation unit (130) calculates an evaluation value for the second artificial neural network model based on the importance of each element of one or more original feature maps for the input data, one or more quantized feature maps for the input data, and one or more original feature maps for the input data.

[0067] According to one embodiment, the evaluation unit (130) may calculate an evaluation value for the second artificial neural network model based on the distance between each original feature map and the quantized feature map corresponding to each original feature map, considering the importance of each element of each original feature map for the input data. Specifically, the evaluation unit (130) may calculate an evaluation value for the second artificial neural network model based on the distance between the result of applying the importance of each element of the feature map generated in the ith layer of the first neural network model among one or more original feature maps to the feature map generated in the ith layer of the first neural network model, and the result of applying the importance of each element of the feature map generated in the ith layer of the first neural network model to the feature map generated in the ith layer of the second neural network model among one or more quantized feature maps.

[0068] For example, the feature map F generated by the i-th layer of the first artificial neural network model among one or more original feature maps for the input data (i) (x) and a feature map generated by the i-th layer of the second artificial neural network model among one or more quantized feature maps for the input data. Distance between Ψ i can be calculated using the mathematical formula 8 below.

[0069] [Equation 8]

[0070]

[0071] In mathematical expression 8, ⊙ represents the elementwise product between matrices, and φ represents a normalization function that normalizes the values ​​of each element in the matrix while preserving the order of the sizes of each element, such as the min-max normalization function or the softmax function. In addition, d2 can be a distance function that can measure the distance between two matrices, such as the matrix norm.

[0072] Meanwhile, according to one embodiment, the evaluation unit (130) may calculate an evaluation value for the second artificial neural network model by applying an aggregation function to the distance between one or more original feature maps and quantized feature maps corresponding to each of the one or more original feature maps. At this time, the aggregation function may be, for example, an average, a weighted sum, a weighted average, etc.

[0073] FIG. 2 is a flowchart of a quantized artificial neural network evaluation method according to one embodiment.

[0074] The method illustrated in FIG. 2 can be performed, for example, by an evaluation device (100).

[0075] Referring to FIG. 2, the evaluation device (100) generates one or more original feature maps for input data using a first artificial neural network model (210).

[0076] Thereafter, the evaluation device (100) determines the importance of each element of one or more original feature maps (220).

[0077] At this time, according to one embodiment, the evaluation device (100) may generate one or more feature maps for each of one or more transformed data for the input data using the first artificial neural network model, and determine the importance of each element of the one or more original feature maps based on the one or more original feature maps and the one or more feature maps for each of the one or more transformed data.

[0078] According to another embodiment, the evaluation device (100) can determine the importance of each element of one or more original feature maps using a metric learning loss function.

[0079] According to another embodiment, the evaluation device (100) can determine the importance for each element of one or more original feature maps using the gradient of each of the one or more original feature maps.

[0080] Thereafter, the evaluation device (100) generates one or more quantized feature maps for the input data using the second artificial neural network model (230).

[0081] At this time, according to one embodiment, the one or more quantized feature maps may be feature maps generated from layers corresponding to each layer of the first neural network model that generated one or more original feature maps among the plurality of layers included in the second artificial neural network model.

[0082] Thereafter, the evaluation device (100) calculates an evaluation value for the second artificial neural network model based on one or more original feature maps, one or more quantized feature maps, and the importance of each element of one or more original feature maps (240).

[0083] At this time, according to one embodiment, the evaluation device (100) can calculate an evaluation value for the second artificial neural network model based on the distance between each original feature map and the quantized feature map corresponding to each original feature map, taking into account the importance of each element of each original feature map for the input data.

[0084] Meanwhile, in the flowchart illustrated in FIG. 2, at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into sub-steps and performed, or one or more steps not illustrated may be added and performed.

[0085] FIG. 3 is a block diagram illustrating a computing environment including a computing device according to one embodiment. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

[0086] The illustrated computing environment (10) includes a computing device (12). The computing device (12) may be one or more components included in an evaluation device (100) according to one embodiment.

[0087] A computing device (12) includes one or more processors (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may cause the computing device (12) to operate according to the exemplary embodiments mentioned above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions, when executed by the processor (14), may be configured to cause the computing device (12) to perform operations according to the exemplary embodiments. Meanwhile, according to one embodiment, the one or more processors (14) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit, but are not necessarily limited thereto.

[0088] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that can be accessed by the computing device (12) and store desired information, or a suitable combination thereof.

[0089] A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

[0090] The computing device (12) may also include one or more input / output interfaces (22) that provide interfaces for one or more input / output devices (24) and one or more network communication interfaces (26). The input / output interfaces (22) and the network communication interfaces (26) are connected to the communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) via the input / output interfaces (22). Exemplary input / output devices (24) may include input devices such as pointing devices (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and / or photographing devices, and / or output devices such as display devices, printers, speakers and / or network cards. The exemplary input / output devices (24) may be included within the computing device (12) as a component constituting the computing device (12), or may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0091] Meanwhile, embodiments of the present invention may include a program for performing the methods described herein on a computer, and a computer-readable recording medium including the program. The computer-readable recording medium may include program commands, local data files, local data structures, etc., alone or in combination. The medium may be specially designed and configured for the present invention, or may be one commonly used in the field of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of the program may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0092] While representative embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims set forth below but also by equivalents thereof.

Claims

1. One or more processors, and A method performed on a computing device having a memory storing one or more programs executed by one or more processors, A step of generating one or more original feature maps for input data using a first artificial neural network model; A step of determining importance for each element of said one or more original feature maps; A step of generating one or more quantized feature maps for the input data using a second artificial neural network model, which is a quantized artificial neural network model for the first artificial neural network model; and A quantized artificial neural network evaluation method, comprising the step of calculating an evaluation value for the second artificial neural network model based on the one or more original feature maps, the one or more quantized feature maps, and the importance.

2. In claim 1, The one or more original feature maps include feature maps generated from one or more layers among the plurality of layers included in the first artificial neural network model, A method for evaluating a quantized artificial neural network, wherein the one or more quantized feature maps include quantized feature maps generated in layers corresponding to each layer of the first artificial neural network model that generated the one or more original feature maps among the plurality of layers included in the second artificial neural network model.

3. In claim 1, A step of generating one or more transformed data for the input data; and Further comprising a step of generating one or more feature maps for each of the one or more transformed data using the first artificial neural network model, A quantized artificial neural network evaluation method, wherein the step of determining the importance determines the importance for each element of the one or more original feature maps based on one or more feature maps for each of the one or more original feature maps and the one or more transformed data.

4. In claim 3, A method for evaluating a quantized artificial neural network, wherein the step of determining the importance determines the importance based on the difference between corresponding elements among each element of each of the one or more original feature maps and each element of each of the one or more transformed data.

5. In claim 1, A method for evaluating a quantized artificial neural network, wherein the step of determining the above importance determines the above importance using a metric learning loss function.

6. In claim 1, A method for evaluating a quantized artificial neural network, wherein the step of determining the importance determines the importance using the gradient of each of the one or more original feature maps.

7. In claim 1, A quantized artificial neural network evaluation method, wherein the step of calculating the evaluation value calculates the evaluation value based on the distance between each of the one or more original feature maps considering the importance and each of the one or more quantized feature maps corresponding to each of the one or more original feature maps.

8. In claim 7, The step of calculating the evaluation value calculates the evaluation value based on the distance between the result of applying the importance of each element of the feature map generated in the ith layer of the first artificial neural network model among the one or more original feature maps to the feature map generated in the ith layer of the first artificial neural network model and the result of applying the importance of each element of the feature map generated in the ith layer of the first artificial neural network model to the feature map generated in the ith layer of the second artificial neural network model among the one or more quantized feature maps.

9. One or more processors; and A memory storing one or more programs executed by said one or more processors, One or more of the above processors, Generate one or more original feature maps for input data using the first artificial neural network model, Determine the importance of each element of one or more of the original feature maps, Generating one or more quantized feature maps for the input data using a second artificial neural network model, which is a quantized artificial neural network model for the first artificial neural network model, A quantized artificial neural network evaluation device that calculates an evaluation value for the second artificial neural network model based on the one or more original feature maps, the one or more quantized feature maps, and the importance.

10. In claim 9, The one or more original feature maps include feature maps generated from one or more layers among the plurality of layers included in the first artificial neural network model, A quantized artificial neural network evaluation device, wherein the one or more quantized feature maps include quantized feature maps generated in layers corresponding to each layer of the first artificial neural network model that generated the one or more original feature maps among the plurality of layers included in the second artificial neural network model.

11. In claim 9, One or more of the above processors, Generate one or more variant data for the above input data, Generating one or more feature maps for each of the one or more transformed data using the first artificial neural network model, A quantized artificial neural network evaluation device that determines the importance of each element of the one or more original feature maps based on one or more feature maps for each of the one or more original feature maps and the one or more transformed data.

12. In claim 11, A quantized artificial neural network evaluation device, wherein said one or more processors determine the importance based on the difference between corresponding elements among each element of said one or more original feature maps and each element of said one or more transformed data.

13. In claim 9, A quantized artificial neural network evaluation device, wherein said one or more processors determine the importance using a metric learning loss function.

14. In claim 9, A quantized artificial neural network evaluation device, wherein said one or more processors determine the importance using the gradient of each of said one or more original feature maps.

15. In claim 9, A quantized artificial neural network evaluation device, wherein the one or more processors calculate the evaluation value based on the distance between each of the one or more original feature maps considering the importance and each of the one or more quantized feature maps corresponding to each of the one or more original feature maps.

16. In claim 15, A quantized artificial neural network evaluation device, wherein the one or more processors calculate the evaluation value based on the distance between the result of applying the importance of each element of the feature map generated in the ith layer of the first artificial neural network model among the one or more original feature maps to the feature map generated in the ith layer of the first artificial neural network model and the result of applying the importance of each element of the feature map generated in the ith layer of the first artificial neural network model to the feature map generated in the ith layer of the second artificial neural network model among the one or more quantized feature maps.

Images