Method and system for quantization-aware training of target model by using ensemble-based knowledge distillation

Abstract

One embodiment of the present invention suggests a method by which at least one processor performs quantization-aware training of a target model by using ensemble-based knowledge distillation. The method comprises the steps of: generating a second candidate model by applying a preset model augmentation technique to first candidate models included in an ensemble candidate group; selecting, from among the second candidate models, a teacher model to be used for training according to the results of a preset evaluation process; calculating, on the basis of a preset training data set and a target model to be trained using the teacher model, a loss function to be used for training the target model; and training the target model by using the loss function.

Description

A method and system for training target model quantization recognition using ensemble-based knowledge distillation.

[0001] The present invention relates to quantization awareness training (QAT) using an ensemble-based knowledge distillation technique for an AI model, and more particularly, to a method and system for selecting a teacher model among candidate models and training a target model using the loss functions of the teacher model and the target model to be trained.

[0002] Quantization-aware training is a technique that predicts the results after quantizing performance evaluation during training to minimize performance degradation that occurs during quantization, and knowledge distillation is a method that improves the performance of a quantized model by setting a model expected to perform well as a teacher model and allowing the quantized model to simulate the judgment results and judgment process of the teacher model.

[0003] Most AI models assume the use of real-valued operations during the learning phase. However, due to issues such as computational speed, power consumption, and heat generation, actual computational environments can only perform a limited number of operations based on input variables with limited bit width.

[0004] In order to perform AI operations on hardware with such a limited operational range, a process is required to convert input / output values ​​and internal coefficient values ​​into data types supported by the hardware.

[0005] In this regard, prior art 1 was presented regarding a support device for neural network operations of fixed-point numbers, which is a generally widely used NPU hardware structure.

[0006] However, prior art 1 has a limitation in that its precision is lower than that of floating point operations generally used in real number representation.

[0007] In this way, the amount of information that can be expressed by data types available for using the NPU hardware structure is significantly less than that of data types generally used to express real numbers. Therefore, when simple conversion is performed, the performance and accuracy of the AI ​​model will be lower than that of a real number-based model due to quantization errors.

[0008] Accordingly, there is a need for a method to improve the performance of AI models by configuring teacher models into an ensemble during the knowledge distillation process and leveraging the strengths of various teacher models.

[0009] (Prior Art 1) Republic of Korea Patent Publication No. 10-2019-0003613, "Support Device and Method for Neural Network Operations on Fixed-Point Numbers with a Relatively Small Number of Digits" (Published January 9, 2019)

[0010] In order to solve the above-described technical problem, the present invention provides a method and system for training target model quantization recognition using ensemble-based knowledge distillation that utilizes the loss functions of a target model to be trained and a teacher model.

[0011] The technical problems to be solved by the present invention are not limited to the technical problems described above, and other technical problems of the present invention can be derived from the following description.

[0012] In order to solve the above-described technical problem, one embodiment of the present invention provides a target model quantization recognition training method using ensemble-based knowledge distillation performed by at least one processor. The method includes the steps of generating a second candidate model by applying a preset model augmentation technique to first candidate models included in an ensemble candidate group, selecting a teacher model to be used for training according to the result of a preset evaluation process among the second candidate models, calculating a loss function to be used for training the target model based on a target model to be trained using the teacher model and a preset learning data set, and training the target model using the loss function.

[0013] In addition, another embodiment of the present invention provides a target model quantization recognition training system using ensemble-based knowledge distillation. The system includes a communication module, at least one processor, and a memory electrically connected to the processor and storing at least one code to be executed by the processor. The memory stores code that, when executed through the processor, causes the processor to generate a second candidate model by applying a preset model augmentation technique to first candidate models included in an ensemble candidate group, select a teacher model to be used for training among the second candidate models based on a result of a preset evaluation process, calculate a loss function to be used for training the target model based on a target model to be trained using the teacher model and a preset learning data set, and train the target model using the loss function.

[0014] According to the solution to the problem of the present invention described above, the performance degradation of quantization recognition training can be minimized.

[0015] In addition, according to the solution to the problem of the present invention described above, the accuracy and training efficiency of the AI ​​model can be improved.

[0016] In addition, according to the solution to the problem of the present invention described above, various candidate models can be generated and evaluated.

[0017] In addition, according to the solution to the problem of the present invention described above, it is possible to derive optimal training performance by updating the weights of the loss function in real time.

[0018] In addition, according to the solution to the problem of the present invention described above, high accuracy and stability for various inputs can be secured simultaneously in the final model.

[0019] In addition, according to the solution to the problem of the present invention described above, computational performance can be maximized in AI acceleration hardware.

[0020] The effects of the present invention are not limited to the effects described above, and include all effects understood from the following description.

[0021] FIG. 1 is a diagram illustrating a detailed configuration of a target model quantization recognition training device using ensemble-based knowledge distillation according to one embodiment of the present invention.

[0022] FIG. 2 is a diagram illustrating a detailed configuration of another example of a target model quantization recognition training device using ensemble-based knowledge distillation according to one embodiment of the present invention.

[0023] FIG. 3 is a diagram illustrating a process in which a target model quantization recognition training device using ensemble-based knowledge distillation illustrated in FIG. 2 trains an artificial intelligence model.

[0024] FIG. 4 is a diagram illustrating in detail the model augmentation and model selection steps during the process of training the artificial intelligence model illustrated in FIG. 3.

[0025] FIG. 5 is a flowchart illustrating the sequence of a target model quantization recognition training method using ensemble-based knowledge distillation according to another embodiment of the present invention.

[0026] FIGS. 6 and 7 are flowcharts illustrating additional steps included in the target model quantization recognition training method using ensemble-based knowledge distillation illustrated in FIG. 5.

[0027] Hereinafter, the present invention will be described in detail with reference to the attached drawings. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In addition, the attached drawings are only intended to facilitate understanding of the embodiments of the invention disclosed herein, and the technical ideas disclosed herein are not limited by the attached drawings. All terms, including technical and scientific terms, used herein should be interpreted as having meanings generally understood by a person of ordinary skill in the art to which the present invention pertains. Terms defined in the dictionary should be interpreted as having additional meanings consistent with the relevant technical literature and the present invention, and shall not be interpreted in an extremely ideal or restrictive sense unless otherwise defined.

[0028] In order to clearly explain the present invention in the drawings, parts irrelevant to the description have been omitted, and the size, shape, and appearance of each component shown in the drawings may be modified in various ways. Identical / similar parts throughout the specification are given identical / similar drawing reference numerals.

[0029] The suffixes "module" and "function" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. Furthermore, in describing the embodiments disclosed herein, detailed descriptions of related known technologies have been omitted if they are deemed to obscure the gist of the embodiments disclosed herein.

[0030] Throughout the specification, when a part is said to be "connected (connected, in contact with, or coupled)" to another part, this includes not only cases where it is "directly connected (connected, in contact with, or coupled)" but also cases where it is "indirectly connected (connected, in contact with, or coupled)" with another member in between. Furthermore, when a part is said to "include (have or provide)" a certain component, this does not mean that it excludes other components, but rather that it may "include (have or provide)" other components, unless otherwise specifically stated.

[0031] As used herein, ordinal terms such as "first," "second," etc., are used solely to distinguish one component from another and do not limit the order or relationship of the components. For example, the first component of the present invention may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component." As used herein, singular expressions should be construed to include plural expressions, unless explicitly stated otherwise.

[0032] FIG. 1 is a diagram illustrating a detailed configuration of a target model quantization recognition training device using ensemble-based knowledge distillation according to one embodiment of the present invention.

[0033] Referring to FIG. 1, a target model quantization recognition training device (100) using ensemble-based knowledge distillation (hereinafter referred to as “target model quantization recognition training device”) may include a communication module (110), a processor (120) that performs operations according to code stored in a memory (130), and a memory (130) that stores the code.

[0034] The target model quantization recognition training device (100) can be implemented as a computer or portable terminal that can connect to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal can include, for example, a wireless communication device that guarantees portability and mobility, and can include all types of handheld-based wireless communication devices such as various types of communication-based terminals, smartphones, tablet PCs, etc. In addition, the portable terminal can be an edge device or on-device AI having at least one processor capable of artificial intelligence model inference work. The network can be implemented as a wired network such as a local area network (LAN), a wide area network (WAN), or a value-added network (VAN), or all types of wireless networks such as a mobile radio communication network or a satellite communication network.

[0035] The communication module (110) may include a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through a wired or wireless connection with other network devices. The communication module transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to technical standards or communication methods for mobile communication used in the mobile communication module (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), etc.).

[0036] The processor (120) may include various types of devices that control and process data. The processor (120) may refer to a data processing device built into hardware that has a physically structured circuit to perform functions expressed by codes or commands included in a program. In one example, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0037] The memory (130) can store at least one of information and data input to the communication module (110), information and data required for functions performed by the processor (120), and data generated according to the execution of the processor (120).

[0038] Memory (130) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain stored information. In addition to a volatile storage device that requires power to maintain stored information, memory (130) may include cloud storage, SSD, magnetic storage media, or flash storage media, but the scope of the present invention is not limited thereto.

[0039] The memory (130) is electrically connected to the processor (120) and stores at least one code that is executed by the processor (120). The memory (130) stores a code that causes the processor (120) to perform the following functions and procedures when executed by the processor (120).

[0040] The memory (130) stores a code that causes the processor (120) to generate a second candidate model by applying a preset model augmentation technique to the first candidate models included in the ensemble candidate group, and to select a teacher model to be used for training among the second candidate models based on the results of a preset evaluation process. The memory (130) stores a code that causes the processor (120) to calculate a loss function to be used for training the target model based on a target model to be trained using the teacher model and a preset learning data set. The memory (130) stores a code that causes the processor (120) to train the target model using the loss function. The evaluation process may include a process of calculating an evaluation score for each of the second candidate models. When selecting a teacher model, the models with the highest evaluation scores N among the second candidate models may be selected as teacher models, or all models with an evaluation score higher than a threshold value S may be selected as teacher models. The maximum number of models N and the threshold value S may be preset values.

[0041] The loss function can be calculated based on the inference results of the teacher model, the inference results of the target model, and the pre-labeling results of the training data set. The loss function can include a first loss function used to train the target model and a second loss function used to train the target model using the teacher model.

[0042] The model augmentation technique may include a pruning technique, a technique applying a model compression algorithm including post-training quantization using a high bit depth to the first candidate models, and a technique applying a transformer layer configured to perform resizing, noise addition, inversion, and inverse transformation on at least one of the input and output ends of each of the first candidate models. For example, if three types of model augmentation techniques are applied to each of the four first candidate models, 12 augmented models are generated, and a second candidate model including a total of 16 models is generated.

[0043] The memory (130) may further store code that causes the processor (120) to assign a first score to each of the second candidate models based on the improvement in accuracy performance of the second candidate models compared to the target model. The memory (130) may further store code that causes the processor (120) to calculate a variance of each of the second candidate models and to assign a second score to each of the second candidate models based on the variance. The memory (130) may further store code that causes the processor (120) to calculate a heterogeneity penalty of each of the second candidate models using a preset formula and to assign a third score to each of the second candidate models based on the heterogeneity penalty. The memory (130) may further store code that causes the processor (120) to derive a ranking for each of the second candidate models based on the first score, the second score, and the third score. The calibration dataset may contain some randomly selected data from the training dataset during initial teacher model selection, and may contain inference results obtained during the training process as actual training progresses.

[0044] The memory (130) may further store code that causes the processor (120) to assign the maximum evaluation score to a model among the second candidate models that has the same model structure as the target model according to preset criteria. For example, if the target model is a real number-based model and there is a real number-based model among the second candidate models, the model is assigned the maximum evaluation score and selected as the teacher model.

[0045] The memory (130) may further store code that causes the processor (120) to define importance coefficients for each of the first loss function and the second loss function, and to calculate a first weight corresponding to the first loss function and a second weight corresponding to the second loss function by normalizing the distribution value of the importance coefficients using a preset formula. The memory (130) may further store code that causes the processor (120) to calculate a final loss function by adding the first loss function and the second loss function. The second loss function may include at least one of a loss function for a final output and a loss function for an intermediate output. The second loss function may include a result of a weighted sum of the loss function for the final output and the loss function for the intermediate output. The importance coefficient may be updated according to a training result during the process of training the target model by the processor (120).

[0046] FIG. 2 is a diagram illustrating a detailed configuration of another example of a target model quantization recognition training device (100) using ensemble-based knowledge distillation according to an embodiment of the present invention. The target model quantization recognition training device (200) using ensemble-based knowledge distillation described below (hereinafter referred to as “target model quantization recognition training device”) has substantially the same configuration as the target model quantization recognition training device (100) described above with reference to FIG. 1. In other words, the operations and functions of the teacher model evaluation unit (210), the loss function calculation unit (220), and the learning unit (230) described below can be performed by the memory (130) and the processor (120) illustrated in FIG. 1. Therefore, redundant descriptions will be omitted.

[0047] Referring to FIG. 2, the target model quantization recognition training device (200) may include a teacher model evaluation unit (210), a loss function calculation unit (220), and a learning unit (230).

[0048] The teacher model evaluation unit (210) can select an appropriate teacher model and derive a list of inference results of the teacher models. The teacher model evaluation unit (210) determines the teacher model to participate in the actual quantization training, and after determining the teacher model, can continuously re-determine the teacher model to participate in the training by repeating it periodically according to the settings. The teacher model evaluation unit (210) can include a model augmentation unit (211) and a model selection unit (212). The model augmentation unit (211) can augment the input candidate teacher model, and the model selection unit (212) can select a teacher model to participate in the actual training based on the augmented model.

[0049] The loss function calculation unit (220) can calculate a loss function by comparing the inference results of the teacher model, the inference results of the training target model, and the labeling results of the training database. The loss function calculation unit (220) can combine and manage the loss function used for basic model training and the loss function used for knowledge distillation.

[0050] The learning unit (230) can perform model training based on the final loss function value calculated by the loss function calculation unit (220).

[0051] Figures 3 and 4 are diagrams illustrating a process in which the target model quantization recognition training device (200) of Figure 2 trains an artificial intelligence model. The operation of the target model quantization recognition training device (200) described below can also be performed in the same manner by the processor (120) and memory (130) of the target model quantization recognition training device (100).

[0052] Referring to FIG. 3, the process of training an artificial intelligence model by a target model quantization recognition training device (200) may include a model augmentation and teacher model selection step (S10), a loss function calculation step (S20), and a target model training step (S30).

[0053] The model augmentation and teacher model selection step (S10) augments the input teacher candidate model and selects a teacher model (500) to participate in actual training based on the augmented teacher candidate model. The model augmentation and teacher model selection step (S10) is described in detail in Fig. 4 below.

[0054] The loss function calculation step (S20) is a step in which the loss function calculation unit (220) calculates the final loss function (800) by combining the loss function used in basic model training and the loss function used in knowledge distillation. The loss function used in basic model training described below has substantially the same configuration as the first loss function (600), and the loss function used in knowledge distillation has substantially the same configuration as the second loss function (700). In the loss function calculation step (S20), the loss function calculation unit (220) can calculate the final loss function (800) by using the sum of two or more loss function values ​​according to the following mathematical expression 1.

[0055]

[0056] In the above mathematical formula 1, is the final loss function (800), The first loss function (600) used for training the quantization model, is the loss function for the final output, is the loss function for the intermediate output, , and represents the weight (C) corresponding to each loss function.

[0057] The second loss function (700) is or It may include one or more of the following: The second loss function (700) and It may include the weighted results. When training a target model against the entire training data set, it can narrow the gap between the inference results of the target model and the labeling results of the training data set. and can narrow the gap between the inference results of the teacher model and the inference results of the target model.

[0058] In order to calculate the weight (C) for each function in the final loss function composition as in the above mathematical expression 1, the importance coefficient for each loss function can be defined and the importance coefficients can be trained. At this time, can be updated according to the training results during the training process of the target model (900). The trained According to mathematical formula 2 The weight (C) of each loss function can be calculated by normalizing based on the distribution value.

[0059]

[0060] In the above mathematical formula 2, is the weight of each loss function x, L is the set of loss functions, represents the importance coefficient of each loss function x.

[0061] The target model training step (S30) is a step in which the learning unit (230) trains the target model (900) using the final loss function (800).

[0062] FIG. 4 is a diagram illustrating in detail the model augmentation and model selection steps during the process of training the artificial intelligence model illustrated in FIG. 3.

[0063] Referring to FIGS. 3 and 4, the model augmentation and teacher model selection step (S10) may include an input teacher candidate model augmentation step (S11) and a teacher model selection step (S12).

[0064] The input teacher candidate model augmentation step (S11) is a step in which the model augmentation unit (211) augments the input teacher candidate model to generate an augmented teacher candidate model. The input teacher candidate model described below has substantially the same configuration as the first candidate model (300), and the augmented teacher candidate model has substantially the same configuration as the second candidate model (400).

[0065] In order for the model augmentation unit (211) to perform model augmentation and generate a second candidate model (400), at least one teacher model candidate group and a list of augmentation techniques may be provided as setting values. The list of augmentation techniques may include all model transformation processes that can be used in algorithm execution without actual training.

[0066] The list of augmentation techniques may include a pruning technique, a technique applying a model compression algorithm including post-training quantization using a high bit-depth, and a technique applying a transformer layer configured to perform resizing, noise addition, inversion, and reverse transformation on at least one of the input and output ends of each of the first candidate models (300).

[0067] The teacher model selection step (S12) is a step in which the model selection unit (212) selects a teacher model (500) used for training the target model (900) from among the second candidate models (400). In order to select the teacher model (500), an evaluation score can be calculated for each model, and the algorithm for calculating the evaluation score is as follows: Steps 1 to 3. Through the following steps 1 to 3, the N models with the highest evaluation scores among the second candidate models (400) can be selected as teacher models (500), or all models with an evaluation score higher than a threshold value S can be selected as teacher models (500). Here, the maximum number of models N and the threshold value S can be preset values. In addition, the maximum evaluation score can be given to a model among the second candidate models (400) having the same model structure as the target model (900) according to preset criteria.

[0068] Step 1. Based on the given calibration dataset, for each of the second candidate models (400), a score is assigned based on the performance improvement of the second candidate model (400) compared to the target model (900) according to the following mathematical equation 3. Here, the calibration dataset may include part or all of the training dataset. In addition, the calibration dataset may include some data randomly selected from the training dataset during the initial teacher model selection, and may include inference results obtained during the training process as the actual training progresses.

[0069]

[0070] In the above mathematical formula 3, is a score based on performance improvement, is the score for score normalization, is the accuracy of the second candidate model (400), refers to the accuracy of the target model (900).

[0071] Process 2. Calculate the dispersion of each second candidate model (400) according to the following mathematical formula 4, and add the calculated dispersion to the score according to the performance improvement calculated in Process 1.

[0072]

[0073] In the above mathematical formula 4, is the dispersion, is a model for the calibration data set A tensor that collects output values ​​from The model is based on the calibration data set A tensor that collects output values ​​from refers to a function that calculates the difference between two tensors.

[0074] Process 3. Based on the input, output, and graph structures of the second candidate model (400) and the target model (900), a heterogeneity penalty is calculated for models among the second candidate models (400) that have high heterogeneity with the target model (900), and a score is given to each of the second candidate models (400) according to the heterogeneity penalty. The heterogeneity penalty is calculated through the cosine similarity with the target model (900) according to the following mathematical expression 5 after converting the layer distribution of each of the second candidate models (400) into a single vector.

[0075]

[0076] In the above mathematical expression 5, is the heterogeneity penalty, is a vector of the set of supported layers, is a layer type within the model The number of, is a vector of the target model (900), Among the second candidate models (400), the layer type is is a vector of the human model.

[0077] An example of a process for training an artificial intelligence model using a target model quantization recognition training device (200) described with reference to FIGS. 3 and 4 is described. Example 1 below is an example applied based on the target model, the YoloX-small model.

[0078] [Example 1]

[0079] A quantization model is generated based on the input model YoloX-small. This is referred to as Model Q below. Then, a set of teacher model candidates is input. Assume there are four of them: DETR, YoloX-small, YoloX-Large, and MobileNet-SSD.

[0080] We perform model augmentation for each. Here, we assume that input image flip, PSNR 30DB AWGN, and 30% pruning are applied. Input image flip creates a model with an additional layer that flips the input left and right and a layer that flips the x-coordinate in the output. PSNR 30DB AWGN adds a noise generation layer to the input and uses the output layer as is. 30% pruning creates a model that applies pruning to the entire model.

[0081] The 12 augmented models are evaluated, and up to four models with scores of 10 or higher are selected. At this time, YoloX-small, which has the same structure as Model Q, is selected as the base teacher model. YoloX-small is referred to as Model F below. The performance of each model is then evaluated to calculate a score. The model performance evaluation scores are shown in Table 1 below.

[0082] Base ScoreVar scoreDifferenceFinal ScoreModel F100--100Model F-flip-0.11.101Model F-AWGN-102.10-7.9Model F-Pruning-202.30-17.7DETR3012.5636.5DETR-flip2812.7634.7DETR-AWGN2513.3632.3DET R-Pruning2011.2625.2YoloX-Large404.2341.2YoloX-Large-flip354.1336.1YoloX-Large-A WGN334.4334.4YoloX-Large-pruning373.7337.7MobileNet-SSD-5.13.24-5.9MobileNet-SSD -flip-6.34.14-6.2MobileNet-SSD-AWGN-6.72.84-7.9MobileNet-SSD-pruning-7.23.14-8.1

[0083] Based on the score results in Table 1 above, four models, Model F, DETR, YoloX-Large, and YoloX-Large-pruning, are selected as teacher models to be used for training. The loss functions are assumed to be three: a training loss function, an output-to-output knowledge distillation function, and an intermediate output-to-output knowledge distillation function. In this case, it is assumed that the importance coefficients R are initialized to (8, 4, 2), respectively. In this case, the weight values ​​of the loss functions are calculated as (1, 0.25, 0.0625), respectively.

[0084] During the training process, the importance coefficient R of the loss function is updated in real time, and the augmented model is re-evaluated at the end of each epoch to re-select the teacher model.

[0085] FIG. 5 is a flowchart illustrating the sequence of a target model quantization recognition training method using ensemble-based knowledge distillation according to another embodiment of the present invention.

[0086] The target model quantization recognition training method using ensemble-based knowledge distillation described below (hereinafter referred to as the “target model quantization recognition training method”) can be performed by the target model quantization recognition training device (100) or the target model quantization recognition training device (200) described above with reference to FIGS. 1 to 4. Therefore, the contents of the embodiments of the present invention described above with reference to FIGS. 1 to 4 can be equally applied to the embodiments to be described below, and any duplicate description with the above description will be omitted below. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps can be performed almost simultaneously.

[0087] Referring to FIG. 5, the target model quantization recognition training method may include a model augmentation application step (S110), a teacher model selection step (S120), a loss function calculation step (S130), and a target model training step (S140).

[0088] The model augmentation application step (S110) is a step of generating a second candidate model by applying a preset model augmentation technique to the first candidate models included in the ensemble candidate group. The model augmentation technique may include a technique of applying a pruning technique to the first candidate models and a transformer layer configured to perform resizing, noise addition, inversion, and reverse transformation on at least one of the input and output terminals of each of the first candidate models.

[0089] The teacher model selection step (S120) is a step for selecting a teacher model to be used for training among the second candidate models based on the results of a preset evaluation process. The evaluation process may include calculating an evaluation score for each of the second candidate models. The teacher model selection step (S120) may include assigning the maximum evaluation score to the model among the second candidate models that has the same model structure as the target model according to preset criteria.

[0090] The loss function calculation step (S130) is a step for calculating a loss function used in training the target model based on a teacher model, a target model to be trained using the teacher model, and a preset learning data set. The loss function may be calculated based on the inference result of the teacher model, the inference result of the target model, and the pre-labeling result of the learning data set. The loss function may include a first loss function used in training the target model and a second loss function used in training the target model using the teacher model.

[0091] The target model training step (S140) is a step for training the target model using a loss function. After the target model training step (S140), a teacher model is reselected, and the loss function calculation step (SS130) and the target model training step (S140) can be repeated for the target model based on the reselected teacher model.

[0092] Referring to FIG. 6, the teacher model selection step (S120) includes a first score assignment step (S121) according to accuracy performance improvement, a second score assignment step (S122) according to variance, a third score assignment step (S123) according to heterogeneity penalty, and a ranking derivation step (S124) for the second candidate models.

[0093] The first score assignment step (S121) according to the accuracy performance improvement is a step of assigning a first score to each of the second candidate models according to the accuracy performance improvement of the second candidate models compared to the target model.

[0094] The second scoring step according to the dispersion (S122) is a step of calculating the dispersion of each of the second candidate models and assigning a second score according to the dispersion to each of the second candidate models.

[0095] Third score assignment step according to heterogeneity penalty (S123) This is a step of calculating the heterogeneity penalty of each of the second candidate models using a preset formula, and assigning a third score according to the heterogeneity penalty to each of the second candidate models.

[0096] The ranking derivation step for the second candidate models (S124) is a step of deriving a ranking for each of the second candidate models based on the first score, the second score, and the third score.

[0097] Referring to FIG. 7, the loss function calculation step (S130) may include an importance coefficient definition step (S131), a weight calculation step (S132), and a final loss function calculation step (S133).

[0098] The importance coefficient definition step (S131) ​​is a step for defining importance coefficients for each of the first loss function and the second loss function. The importance coefficients may be updated according to the training results during the training process of the target model in the target model training step (S140).

[0099] The weight calculation step (S132) is a step of calculating a first weight corresponding to the first loss function and a second weight corresponding to the second loss function by standardizing based on the distribution value of the importance coefficient using a preset formula.

[0100] The final loss function calculation step (S133) is a step of calculating the final loss function by adding the first loss function and the second loss function.

[0101] The target model quantization recognition training method of the embodiments of the present invention described so far can also be implemented in the form of a recording medium including computer-executable instructions, such as program modules executed by a computer. The computer-readable medium can be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium can include a computer storage medium. The computer storage medium includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0102] Those skilled in the art will appreciate that the present invention can be easily modified into other specific forms without changing the technical spirit or essential characteristics of the present invention based on the above description. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention. The scope of the present application is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present application.

[0103] The form for carrying out the invention is substantially the same as the best form for carrying out the invention.

[0104] The present invention can be used in a technology for minimizing performance degradation during model quantization, and thus has industrial applicability.

Claims

1. A method for training a target model quantization recognition using ensemble-based knowledge distillation performed by at least one processor, a) A step of creating a second candidate model by applying a preset model augmentation technique to the first candidate models included in the ensemble candidate group; b) a step of selecting a teacher model to be used for training based on the results of a preset evaluation process among the second candidate models; c) a step of calculating a loss function used for training the target model based on a target model to be trained using the teacher model and a preset learning data set; and d) a step of training the target model using the loss function, A method for training target model quantization recognition using ensemble-based knowledge distillation.

2. In paragraph 1, The above loss function is, It is calculated based on the inference result of the above teacher model, the inference result of the above target model, and the pre-labeling result of the above learning data set. A method for training target model quantization recognition using ensemble-based knowledge distillation.

3. In paragraph 1, The above model augmentation technique is, A technique including applying a pruning technique to the first candidate models and a transformer layer configured to perform resizing, noise addition, inversion, and reverse transformation on at least one of the input and output terminals of each of the first candidate models. A method for training target model quantization recognition using ensemble-based knowledge distillation.

4. In paragraph 1, The above evaluation process is, A process for calculating an evaluation score for each of the second candidate models is included. A method for training target model quantization recognition using ensemble-based knowledge distillation.

5. In paragraph 4, Step b) above, b-1) A step of assigning a first score to each of the second candidate models according to the improvement in accuracy performance of the second candidate models compared to the target model; b-2) A step of calculating the dispersion of each of the second candidate models and assigning a second score according to the dispersion to each of the second candidate models; b-3) A step of calculating a heterogeneity penalty for each of the second candidate models using a preset formula, and assigning a third score according to the heterogeneity penalty to each of the second candidate models; and b-4) A step of deriving a ranking for each of the second candidate models based on the first score, the second score, and the third score, A method for training target model quantization recognition using ensemble-based knowledge distillation.

6. In paragraph 4, Step b) above, It further includes a step of assigning the maximum value of the evaluation score to a model among the second candidate models having the same model structure as the target model according to preset criteria. A method for training target model quantization recognition using ensemble-based knowledge distillation.

7. In paragraph 1, The above loss function is, A first loss function used for training the target model and a second loss function used for training the target model using the teacher model are included. A method for training target model quantization recognition using ensemble-based knowledge distillation.

8. In paragraph 7, Step c) above, c-1) A step of defining importance coefficients for each of the first loss function and the second loss function; c-2) A step of calculating a first weight corresponding to the first loss function and a second weight corresponding to the second loss function by normalizing the distribution value of the importance coefficient using a preset formula; and c-3) A step of calculating a final loss function by adding the first loss function and the second loss function, A method for training target model quantization recognition using ensemble-based knowledge distillation.

9. In paragraph 8, The above importance coefficient is, In the training process of the target model of step d), it is updated according to the training result. A method for training target model quantization recognition using ensemble-based knowledge distillation.

10. Communication module; at least one processor; and A memory electrically connected to the processor and storing at least one code to be executed by the processor, The above memory, when executed through the above processor, The processor generates a second candidate model by applying a preset model augmentation technique to the first candidate models included in the ensemble candidate group, selects a teacher model to be used for training among the second candidate models based on the results of a preset evaluation process, calculates a loss function to be used for training the target model based on a target model to be trained using the teacher model and a preset learning data set, and stores a code that causes the target model to be trained using the loss function. A target model quantization recognition training system using ensemble-based knowledge distillation.

11. In paragraph 10, The above loss function is, It is calculated based on the inference result of the above teacher model, the inference result of the above target model, and the pre-labeling result of the above learning data set. A target model quantization recognition training system using ensemble-based knowledge distillation.

12. In paragraph 10, The above model augmentation technique is, A technique including applying a pruning technique to the first candidate models and a transformer layer configured to perform resizing, noise addition, inversion, and reverse transformation on at least one of the input and output terminals of each of the first candidate models. A target model quantization recognition training system using ensemble-based knowledge distillation.

13. In paragraph 10, The above evaluation process is, A process for calculating an evaluation score for each of the second candidate models is included. A target model quantization recognition training system using ensemble-based knowledge distillation.

14. In paragraph 13, The above memory is, The processor further stores code that causes the processor to assign a first score to each of the second candidate models according to an accuracy performance improvement of the second candidate models compared to the target model, calculate a variance of each of the second candidate models, assign a second score to each of the second candidate models according to the variance, calculate a heterogeneity penalty of each of the second candidate models using a preset formula, assign a third score to each of the second candidate models according to the heterogeneity penalty, and derive a ranking for each of the second candidate models based on the first score, the second score, and the third score. A target model quantization recognition training system using ensemble-based knowledge distillation.

15. In paragraph 13, The above memory is, The processor further stores a code that causes the second candidate models to assign the maximum value of the evaluation score to a model having the same model structure as the target model according to preset criteria. A target model quantization recognition training system using ensemble-based knowledge distillation.

16. In paragraph 10, The above loss function is, A first loss function used for training the target model and a second loss function used for training the target model using the teacher model are included. A target model quantization recognition training system using ensemble-based knowledge distillation.

17. In paragraph 16, The above memory is, The processor further stores code that causes the processor to define importance coefficients for each of the first loss function and the second loss function, normalize the importance coefficients based on a distribution value of the importance coefficients using a preset formula, calculate a first weight corresponding to the first loss function and a second weight corresponding to the second loss function, and calculate a final loss function by combining the first loss function and the second loss function. A target model quantization recognition training system using ensemble-based knowledge distillation.

18. In paragraph 17, The above importance coefficient is, The above processor is updated according to the training results during the process of training the target model. A target model quantization recognition training system using ensemble-based knowledge distillation.

Images