Method of training deep learning model using self-knowledge distillation algorithm and deep learning model training device performing method

Abstract

A method for training a deep learning model using a self-knowledge distillation algorithm according to an embodiment of the present invention may include: a step of inputting training data into a deep learning model corresponding to a first time point to obtain a first output vector, and inputting the training data into a deep learning model corresponding to a second time point prior to the first time point to obtain a second output vector; a step of generating a soft target vector at a first time point for the training data using the second output vector and the correct answer data; a step of aligning the first output vector and the soft target vector, and generating a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector; and a step of training the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution.

Description

Technology Field

[0001] The present invention relates to a method for training a deep learning model using a self-knowledge distillation algorithm and a training device for a deep learning model that performs the method. Background Technology

[0002] It is easy to train deep learning models effectively in fields where a large amount of data is available. However, in fields where data is scarce, it is difficult to train deep learning models effectively, and recently, knowledge distillation through transfer learning has been attracting attention to solve this problem.

[0003] Knowledge distillation can refer to the process of transferring knowledge from a well-trained large network (teacher network) to a small network (student network) that intends to actually use it.

[0004] In this regard, the prior art discloses a self-knowledge distillation technique that utilizes the prediction results of a deep learning model corresponding to the previous number of training iterations during the training process of a deep learning model. However, there was a limitation in that knowledge regarding low confidence was ignored because there was a significant difference between the value with the highest confidence (or prediction certainty) and the value with the lowest confidence in the prediction results.

[0005] Therefore, there is a need to develop a method to improve the performance of deep learning models by learning without ignoring knowledge about low confidence in self-knowledge distillation techniques. Prior art literature

[0006] self-knowledge distillation with progressive refinement of targets, ICCV 2021, https: / / arxiv.org / abs / 2006.12000 The problem to be solved

[0007] The problem to be solved by the present invention is to provide a method for training a deep learning model using a self-knowledge distillation algorithm, wherein training data is input into a deep learning model corresponding to a first time point to obtain a first output vector, the training data is input into a deep learning model corresponding to a second time point prior to the first time point to obtain a second output vector, a soft target vector for the training data at the first time point is generated using the second output vector and correct answer data, the first output vector and the soft target vector are aligned, a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector are generated, and the deep learning model is trained to minimize a first loss function determined based on the first partial distribution and the second partial distribution.

[0008] However, the problems that the present invention aims to solve are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0009] A method for training a deep learning model according to an embodiment of the present invention may include: a step of inputting training data into a deep learning model corresponding to a first time point to obtain a first output vector, and inputting the training data into a deep learning model corresponding to a second time point prior to the first time point to obtain a second output vector; a step of generating a soft target vector at a first time point for the training data using the second output vector and the correct answer data; a step of aligning the first output vector and the soft target vector, and generating a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector; and a step of training the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution.

[0010] Here, the number of training sessions of the deep learning model corresponding to the first time point and the number of training sessions of the deep learning model corresponding to the second time point may be different.

[0011] Additionally, the step of generating the first partial distribution and the second partial distribution may include: a step of aligning the soft target vector based on a confidence score regarding multiple classifications; and a step of aligning the first output vector in the same class order as the aligned soft target vector.

[0012] Additionally, the step of generating the first partial distribution and the second partial distribution may include the step of generating the first partial distribution and the second partial distribution by dividing the entire class included in the first output vector and the soft target vector into a predetermined number of classes.

[0013] Additionally, the above method may further include the step of generating a first partial probability distribution and a second partial probability distribution using a softmax function from the first partial distribution and the second partial distribution.

[0014] Here, the first loss function can be determined based on the difference between the first partial probability distribution and the second partial probability distribution.

[0015] Additionally, the step of training the deep learning model may further include: a step of determining a second loss function based on the total distribution of the first output vector and the soft target vector; and a step of training the deep learning model to minimize a third loss function, which is a weighted sum of the first loss function and the second loss function.

[0016] A training device for a deep learning model using a self-knowledge distillation algorithm according to another embodiment of the present invention includes: a memory in which a deep learning model training program is stored; and a processor that loads the deep learning model training program from the memory and executes the deep learning model training program. The processor can input training data into a deep learning model corresponding to a first time point to obtain a first output vector, input the training data into a deep learning model corresponding to a second time point prior to the first time point to obtain a second output vector, generate a soft target vector for the training data at the first time point using the second output vector and the correct answer data, align the first output vector and the soft target vector, generate a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector, and train the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution. Effects of the invention

[0017] According to an embodiment of the present invention, the present invention can achieve the effect of improving the performance of a deep learning model by training the deep learning model to reinforce learning about knowledge regarding high-confidence classes and not ignore knowledge regarding low-confidence classes, without incurring a significant cost burden for additional models or additional computational loads for learning.

[0018] In addition, according to an embodiment of the present invention, the present invention is a plug-in method, and since the performance of a deep learning model can be improved without changing the existing self-knowledge distillation algorithm through the present invention, the effect of reducing the development cost of the deep learning model can be achieved. Brief explanation of the drawing

[0019] FIG. 1 is a block diagram showing a deep learning model training device according to an embodiment of the present invention. FIG. 2 is a block diagram conceptually illustrating the functions of a deep learning model training program according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a deep learning model training method according to an embodiment of the present invention. FIG. 4 is a flowchart showing the process of determining a first loss function according to one embodiment of the present invention. FIG. 5 is a diagram illustrating a method for training a deep learning model based on a first loss function and a second loss function determined according to an embodiment of the present invention. Specific details for implementing the invention

[0020] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0021] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions in the embodiments of the present invention, and these definitions may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0022] FIG. 1 is a block diagram showing a deep learning model training device according to an embodiment of the present invention.

[0023] Referring to FIG. 1, a deep learning model training device (100) may include a processor (110), an input / output device (120), and a memory (130).

[0024] The processor (110) can control the overall operation of the deep learning model training device (100).

[0025] The processor (110) can receive training data and correct answer data using an input / output device (120). Here, the training data and correct answer data according to one embodiment of the present invention may refer to data for multi-classification. For example, the training data may include images for classifying objects (e.g., cars, cats, etc.), and the correct answer data may include data labeled for objects (e.g., cars, cats, etc.) within the images.

[0026] In the present invention, training data and correct answer data are described as being input through an input / output device (120), but are not limited thereto. That is, according to an embodiment, the deep learning model training device (100) may include a transceiver (not shown), and the deep learning model training device (100) may receive at least one of the training data and correct answer data using the transceiver (not shown), and at least one of the training data and correct answer data may be generated within the deep learning model training device (100).

[0027] The processor (110) can improve the performance of the deep learning model by inputting training data into a deep learning model corresponding to a first time point to obtain a first output vector, inputting the training data into a deep learning model corresponding to a second time point prior to the first time point to obtain a second output vector, generating a soft target vector for the training data at the first time point using the second output vector and the correct answer data, aligning the first output vector and the soft target vector, generating a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector, and training the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution.

[0028] The input / output device (120) may include one or more input devices and / or one or more output devices. For example, the input device may include a microphone, keyboard, mouse, touch screen, etc., and the output device may include a display, speaker, etc.

[0029] The memory (130) can store information necessary for the execution of the deep learning model training program (200) and the deep learning model training program (200).

[0030] In the present specification, the deep learning model training program (200) may refer to software that receives training data and correct answer data as input and includes instructions for improving the performance of a deep learning model using a self-knowledge distillation algorithm.

[0031] The processor (110) can load the deep learning model training program (200) and information necessary for the execution of the deep learning model training program (200) from memory (130) in order to execute the deep learning model training program (200).

[0032] The processor (110) can improve the performance of a deep learning model using a self-knowledge distillation algorithm by executing a deep learning model training program (200).

[0033] The functions and / or operations of the deep learning model training program (200) will be examined in detail through FIG. 2.

[0034] FIG. 2 is a block diagram conceptually illustrating the functions of a deep learning model training program according to an embodiment of the present invention.

[0035] Referring to FIG. 2, the deep learning model training program (200) may include an output vector acquisition unit (210), a soft target vector generation unit (220), a partial distribution generation unit (230), and a deep learning model training unit (240).

[0036] The output vector acquisition unit (210), soft target vector generation unit (220), partial distribution generation unit (230), and model learning unit (240) illustrated in FIG. 2 are conceptual divisions of the functions of the deep learning model learning program (200) to easily explain the functions of the deep learning model learning program (200), but are not limited thereto. According to embodiments, the functions of the output vector acquisition unit (210), soft target vector generation unit (220), partial distribution generation unit (230), and deep learning model learning unit (240) can be merged or separated and may be implemented as a series of instructions included in a single program.

[0037] First, the output vector acquisition unit (210) can acquire a first output vector by inputting the training data into a deep learning model corresponding to a first time point, and acquire a second output vector by inputting the training data into a deep learning model corresponding to a second time point prior to the first time point.

[0038] Here, a deep learning model according to one embodiment of the present invention may refer to a deep learning model that uses a known self-knowledge distillation technique.

[0039] Specifically, the first output vector and the second output vector according to one embodiment of the present invention are inputs to and outputs deep learning models corresponding to the first time point and the second time point, respectively, and may represent a distribution regarding confidence (class probability or prediction confidence) regarding the classification of given classes.

[0040] Meanwhile, the number of training sessions of a deep learning model corresponding to a first time point according to one embodiment of the present invention and the number of training sessions of a deep learning model corresponding to a second time point may be different from each other. For example, the number of training sessions (epochs) of a deep learning model corresponding to a first time point may be 10, and the number of training sessions of a deep learning model corresponding to a second time point prior to the first time point may be 5.

[0041] Through this, when training a deep learning model at the first time point, the deep learning model at the second time point is used, thereby achieving the effect of improving the performance of the deep learning model without using an additional model such as a teacher model.

[0042] Next, the soft target vector generation unit (220) can generate a soft target vector at a first time point for the training data using the second output vector and the correct answer data.

[0043] Specifically, a soft target vector generation unit (220) according to one embodiment of the present invention is a correct answer label ( ) and the second output vector( By performing a weighted sum operation on ), the soft target vector ( ) can be generated. Here, the above soft target vector ( ) can be determined using the following mathematical formula 1.

[0045]

[0047] Here, can mean the soft target vector at the first time point for the training data, and can mean a value between 0 and 1, and can mean the correct label of the correct answer data, and may mean a second output vector that is output by inputting the training data into a deep learning model corresponding to the second time point.

[0049] Next, the partial distribution generation unit (230) can align the first output vector and the soft target vector, and generate a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector.

[0050] Specifically, a partial distribution generation unit (230) according to one embodiment of the present invention can align the soft target vector based on a confidence score regarding multi-class classification. Here, the confidence score according to one embodiment of the present invention may mean a score expressed based on normalization of the confidence (or probability value, prediction confidence) regarding the classification of given classes.

[0051] For example, the partial distribution generation unit (230) can sort the soft target vector in descending or ascending order of confidence by referring to the confidence score of each class included in the soft target vector.

[0052] In addition, the partial distribution generation unit (230) according to one embodiment of the present invention can align the first output vector in the same order as the class order of the aligned soft target vector.

[0053] For example, the partial distribution generation unit (230) can sort the first output vector in the same order as the class order of the soft target vectors sorted in descending or ascending order of confidence.

[0054] Meanwhile, a partial distribution generation unit (230) according to one embodiment of the present invention can generate a first partial distribution and a second partial distribution by dividing the entire class included in the first output vector and the soft target vector into a predetermined number of classes.

[0055] Specifically, a partial distribution generation unit (230) according to one embodiment of the present invention can generate at least one first partial distribution and at least one second partial distribution (e.g., the number of first partial distributions and the number of second partial distributions are each 20) by using a window having a specific size to divide the entire class included in the first output vector and the soft target vector (e.g., the total number of classes in the first output vector and the soft target vector is 100 each) by a preset number of classes (e.g., the preset number of classes is 5).

[0056] Meanwhile, the number of classes set to generate the first and second partial distributions is not limited to the above examples and can be varied in various ways to achieve the objectives of the present invention.

[0057] In addition, a partial distribution generation unit (230) according to one embodiment of the present invention can generate a first partial probability distribution and a second partial probability distribution using a softmax function from the first partial distribution and the second partial distribution.

[0058] Specifically, the partial distribution generation unit (230) can generate at least one first partial probability distribution using a softmax function for at least one first partial distribution, and can generate at least one second partial probability distribution using a softmax function for at least one second partial distribution.

[0059] Through this, in the learning process of the deep learning model, as partial distributions regarding classes distinguished by confidence scores are generated as partial probability distributions having values ​​between 0 and 1 regardless of confidence scores through a softmax function, the first partial distribution and the second partial distribution regarding the class corresponding to the low confidence score in the soft output vector are considered identically to the first partial distribution and the second partial distribution regarding the class corresponding to the high confidence score in the soft output vector, thereby achieving the effect of learning without ignoring knowledge corresponding to the low confidence score.

[0060] Next, the deep learning model training unit (240) can train the deep learning model to minimize the first loss function determined based on the first partial distribution and the second partial distribution.

[0061] Specifically, a first loss function according to one embodiment of the present invention may be determined based on the difference (or distance) between a first partial probability distribution generated from a first partial distribution and a second partial probability distribution generated from a second partial distribution. For example, the first loss function may mean a loss function based on Kullback-Leibler divergence (KL divergence) or cross-entropy.

[0062] In addition, a deep learning model training unit (240) according to one embodiment of the present invention can determine a second loss function based on the total distribution of the first output vector and the soft target vector, and train the deep learning model to minimize a third loss function which is a weighted sum of the first loss function and the second loss function.

[0063] Specifically, a deep learning model training unit (240) according to one embodiment of the present invention can determine a second loss function (e.g., the second loss function is a distillation loss function used in a conventional self-knowledge distillation algorithm) based on the difference (or distance) between the total distribution of the first output vector and the total distribution of the soft target vector, and can train a deep learning model to minimize a third loss function which is a weighted sum of the first loss function and the second loss function.

[0064] Here, the third loss function can be expressed as Equation 2 below.

[0066]

[0068] Here, can mean the third loss function, and may refer to the second loss function, which is the distillation loss function used in conventional self-knowledge distillation algorithms, and can mean the first loss function, and can mean a hyperparameter with a value between 0.01 and 0.1.

[0070] Through this, by training the deep learning model to reinforce learning about high-confidence classes and not ignore knowledge about low-confidence classes, the effect of improving the performance of the deep learning model can be achieved.

[0072] FIG. 3 is a flowchart illustrating a deep learning model training method according to an embodiment of the present invention.

[0073] Referring to FIGS. 2 and 3, a processor (110) according to one embodiment of the present invention may input training data into a deep learning model corresponding to a first time point to obtain a first output vector, and input the training data into a deep learning model corresponding to a second time point prior to the first time point to obtain a second output vector (S310).

[0074] Next, the processor (110) can generate a soft target vector at a first time point for the training data using the second output vector and the correct answer data (S320).

[0075] Then, the processor (110) may align the first output vector and the soft target vector, and generate a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector (S330).

[0076] Then, the processor (110) can perform training on the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution (S340).

[0078] FIG. 4 is a flowchart showing the process of determining a first loss function according to one embodiment of the present invention.

[0079] Referring to FIGS. 2 and FIGS. 4, a processor (110) according to one embodiment of the present invention may align soft target vectors based on a confidence score regarding multiple classifications (S410), and align a first output vector in the same class order as the aligned soft target vectors (S420).

[0080] Then, the processor (110) can generate a first partial distribution and a second partial distribution by dividing the entire class included in the first output vector and the soft target vector into a preset number of classes (S430), generate a first partial probability distribution and a second partial probability distribution using a softmax function from the first partial distribution and the second partial distribution (S440), and determine a first loss function based on the difference between the first partial probability distribution and the second partial probability distribution (S450).

[0082] FIG. 5 is a diagram illustrating a method for training a deep learning model based on a first loss function and a second loss function determined according to an embodiment of the present invention.

[0083] Referring to FIGS. 2 and FIGS. 5, a processor (110) according to one embodiment of the present invention can input a training image into a deep learning model corresponding to a first time point to obtain a first output vector (510). Additionally, the processor (110) can input a training image into a deep learning model corresponding to a second time point prior to the first time point to obtain a second output vector, and based on a self-knowledge distillation algorithm, can generate a soft target vector (520) for the training data at the first time point using the second output vector and the correct image.

[0084] Meanwhile, a processor (110) according to one embodiment of the present invention may first sort soft target vectors (520) in descending order of reliability based on reliability scores regarding multiple classifications. Additionally, the processor (110) may sort a first output vector (510) in the same class order as the sorted soft target vectors (521). Here, since the sorted first output vector (511) is not sorted based on reliability scores, it can be seen that the distribution of the sorted first output vector (511) is not smooth, unlike the sorted soft target vector (521).

[0085] In addition, a processor (110) according to one embodiment of the present invention can generate at least one first partial distribution and at least one second partial distribution by using a window of a specific size to divide the entire class included in the first output vector and the soft target vector into a predetermined number of classes. In addition, the processor (110) can generate at least one first partial probability distribution (512) using a softmax function for at least one first partial distribution, and can generate at least one second partial probability distribution (522) using a softmax function for at least one second partial distribution.

[0086] Additionally, a processor (110) according to one embodiment of the present invention can determine a first loss function (530) based on the difference (or distance) between at least one first partial probability distribution (512) and at least one second partial probability distribution (522).

[0087] Meanwhile, a processor (110) according to one embodiment of the present invention can determine a second loss function (540) based on the difference (or distance) between the total distribution (510) of the first output vector and the total distribution (520) of the soft target vector, and can train a deep learning model by backpropagating to minimize a third loss function, which is a weighted sum of the first loss function and the second loss function. Here, according to one embodiment of the present invention, the third loss function may be given a greater weight to the second loss function than to the first loss function.

[0088] Through this, the effect of improving the performance of the model can be achieved by training the deep learning model to reinforce learning about knowledge regarding high-confidence classes and not ignore knowledge regarding low-confidence classes, without the significant cost burden of additional models or additional computational load for learning.

[0089] Combinations of each block of the block diagram attached to the present invention and each step of the flowchart may be performed by computer program instructions. Since these computer program instructions may be loaded into an encoding processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions performed through the encoding processor of the computer or other programmable data processing equipment create means for performing the functions described in each block of the block diagram or each step of the flowchart. Since these computer program instructions may also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory may also produce a manufactured item containing instruction means for performing the function described in each block of the block diagram or each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0090] Additionally, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps described in succession may actually be performed substantially simultaneously, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

[0091] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0093] 100: Deep learning model training device 200: Deep learning model training program 210: Data Selection Unit 220: Soft target vector generation unit 230: Partial distribution generation unit 240: Deep learning model training unit

Claims

Claim 1 A method for training a deep learning model using a self-knowledge distillation algorithm, comprising: a step of inputting training data into the deep learning model at a first time point to obtain a first output vector, and inputting the training data into the deep learning model at a second time point prior to the first time point to obtain a second output vector; a step of generating a soft target vector at the first time point for the training data by performing a weighted sum operation on the second output vector and the correct answer data; and a step of aligning the first output vector and the soft target vector, and generating a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector. and a step of training the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution; wherein the number of training iterations of the deep learning model at the first time point is greater than the number of training iterations of the deep learning model at the second time point; and the step of training the deep learning model further includes: a step of determining a second loss function based on the total distribution of the first output vector and the soft target vector; and a step of training the deep learning model to minimize a third loss function obtained by the weighted sum of the first loss function and the second loss function; wherein the third loss function is a mathematical formula Expressed as, and the above is the above third loss function, above is the above second loss function, above is the above first loss function, above A deep learning model training method in which each hyperparameter has a value between 0.01 and 0.

1. Claim 2 delete Claim 3 A deep learning model training method according to claim 1, wherein the step of generating the first partial distribution and the second partial distribution comprises: the step of aligning the soft target vector based on a confidence score regarding multi-classification; and the step of aligning the first output vector in the same class order as the aligned soft target vector. Claim 4 A deep learning model training method according to claim 1, wherein the step of generating the first partial distribution and the second partial distribution comprises the step of generating the first partial distribution and the second partial distribution by dividing the entire class included in the first output vector and the soft target vector into a predetermined number of classes. Claim 5 A deep learning model training method according to claim 1, further comprising the step of generating a first partial probability distribution and a second partial probability distribution using a softmax function from the first partial distribution and the second partial distribution. Claim 6 A deep learning model training method according to claim 4, wherein the first loss function is determined based on the difference between the first partial probability distribution and the second partial probability distribution. Claim 7 delete Claim 8 A training device for a deep learning model using a self-knowledge distillation algorithm, comprising a memory in which a deep learning model training program is stored; The system includes a processor that loads the deep learning model training program from the memory and executes the deep learning model training program, wherein the processor inputs training data into the deep learning model at a first time point to obtain a first output vector, inputs the training data into the deep learning model at a second time point prior to the first time point to obtain a second output vector, generates a soft target vector at the first time point for the training data by performing a weighted sum operation on the second output vector and the correct answer data, aligns the first output vector and the soft target vector, generates a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector, and trains the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution, wherein the number of training iterations of the deep learning model at the first time point is greater than the number of training iterations of the deep learning model at the second time point, and wherein the processor [determines] the second loss function based on the total distribution of the first output vector and the soft target vector Determine and train the deep learning model to minimize a third loss function, which is a weighted sum of the first loss function and the second loss function, wherein the third loss function is a mathematical formula Expressed as, and the above is the above third loss function, above is the above second loss function, above is the above first loss function, above A deep learning model training device that represents hyperparameters having values ​​between 0.01 and 0.1, respectively. Claim 9 delete Claim 10 In claim 8, the processor aligns the soft target vectors based on a confidence score regarding multi-classification and aligns the first output vectors in the same class order as the aligned soft target vectors, a deep learning model training device. Claim 11 In claim 8, the processor is a deep learning model training device that generates the first partial distribution and the second partial distribution by dividing the entire class included in the first output vector and the soft target vector into a predetermined number of classes. Claim 12 In claim 8, the processor is a deep learning model training device that generates a first partial probability distribution and a second partial probability distribution using a softmax function from the first partial distribution and the second partial distribution. Claim 13 A deep learning model training device in which, in claim 12, the first loss function is determined based on the difference between the first partial probability distribution and the second partial probability distribution. Claim 14 delete Claim 15 A computer-readable recording medium storing a computer program, comprising: a step of inputting training data into a deep learning model at a first time point to obtain a first output vector, and inputting the training data into the deep learning model at a second time point prior to the first time point to obtain a second output vector; a step of generating a soft target vector at the first time point for the training data by performing a weighted sum operation on the second output vector and the correct answer data; a step of aligning the first output vector and the soft target vector, and generating a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector; and a step of training the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution; wherein the number of training iterations of the deep learning model at the first time point is greater than the number of training iterations of the deep learning model at the second time point, and the step of training the deep learning model comprises a step of determining a second loss function based on the total distribution of the first output vector and the soft target vector. and further comprising the step of training the deep learning model to minimize a third loss function, which is a weighted sum of the first loss function and the second loss function; wherein the third loss function is a mathematical formula Expressed as, and the above is the above third loss function, above is the above second loss function, above is the above first loss function, above A computer-readable recording medium comprising instructions for a processor to perform a training method of a deep learning model using a self-knowledge distillation algorithm, each meaning a hyperparameter having a value between 0.01 and 0.

1. Claim 16 A computer program stored on a computer-readable recording medium, comprising: a step of inputting training data into a deep learning model at a first time point to obtain a first output vector, and inputting the training data into the deep learning model at a second time point prior to the first time point to obtain a second output vector; a step of generating a soft target vector at the first time point for the training data by performing a weighted sum operation on the second output vector and the correct answer data; a step of aligning the first output vector and the soft target vector, and generating a first partial distribution regarding the aligned first output vector and a second partial distribution regarding the aligned soft target vector; and a step of training the deep learning model to minimize a first loss function determined based on the first partial distribution and the second partial distribution; wherein the number of training iterations of the deep learning model at the first time point is greater than the number of training iterations of the deep learning model at the second time point, and the step of training the deep learning model comprises a step of determining a second loss function based on the total distribution of the first output vector and the soft target vector. and further comprising the step of training the deep learning model to minimize a third loss function, which is a weighted sum of the first loss function and the second loss function; wherein the third loss function is a mathematical formula Expressed as, and the above is the above third loss function, above is the above second loss function, above is the above first loss function, above A computer program comprising instructions for a processor to perform a training method of a deep learning model using a self-knowledge distillation algorithm, each representing a hyperparameter having a value between 0.01 and 0.1.

Images